Some of it is the usual low-grade irritation inspired by advertising of most sorts, I'm sure. But billboards don't throw me into a rage, nor even does the paper junk mail I get fairly regularly.
I think what I really loathe about spam is the dishonesty of so much of it. The desperately pathetic attempt to get me to open the message by pretending to be from a friend, by masquerading as a message from someone I might actually give a damn about. For the most part, junk mail is easily identifiable, it doesn't pretend to be a handwritten letter from my mother, for example.
It makes me wonder "how stupid do you think I am?" which of course is a major trigger of mine - I don't sniff boots, but don't call me stupid, in any case. So the more spam I get, the more I feel like I'm being called stupid, I guess. So I get crankier and crankier as I delete my way through my folders...
I'd have to admit, yes, that I find the imprecation against my intelligence more annoying than the disgusting subject lines of some of the worst offenders.