Warning: Long Diatribe is Long.


Although my nominal profession is web developer, I end up doing most
of the technical service. If the server needs reconfiguration, I do
it. If someone's PC catches fire, I have to smother it with blankets.
I write the automation scrips to backup the server. Fair enough,
it's a small company and I was the first person there who could tell
the difference between 'cp' and 'rm'.

The one true burr under my saddle is having to deal with email. It
isn't so bad if people are screaming in the event of an outage, and
it's only mildly annoying when they forget their passwords. But what
drives me most insane is deliverability issues. I hate staring tensely at mail queues, surfing through maillog, and poking qmail to retry all those queued messages in a reasonable timeframe. It's clear: ISPs have
no business doing server-side blocking of junk mail.

1. They don't know what spam is.

As good as filtering technology has become, no system is absolutely
correct. This is both because filters don't understand content, and
because they don't understand users or their intended uses. A
legitimate bank statement looks remarkably like a quality phishing
job. That perfectly formatted gibberish sent to ten addresses? It
was a $1200 e-commerce order being forwarded to appropriate
departments for fulfillment. And sadly, for some people, the
mis-spelled Viagra solicitation from a site in Hong Kong is
desperately awaited.

It's further complicated by the fact even users don't know what spam
is. I'm sure you-- or someone you know-- has simply used the "file as
spam" button to make a once-requested newsletter go away. The whole
"never, ever click a 'remove' link" indoctrination hardly helps that
particular problem. Who gets penalized? The list issuer, who may
well have gone to great lengths for compliance.

Furthermore, there are often legitimate times to pass spam. Many
users, for example, appeciate a "branded" email address, but want it
to all be forwarded to their ISP address. It would be impolite, if
not downright negligent, to try to go through and delete what we think
is spam before it was passed on.

2. These filters tend to set their own rules. I note several of the
large mail providers-- unsurprisingly the ones with the most obnoxious
filtering problems-- seem to be all to happy to get you to sign up for
white-list service. This is an disaster in many distinct ways:

- They frequently demand behaviour well beyond compliance with anti-spam laws.

- They tend to be organized only in the concept of two classes of
users. You're either a small corporation who never has any business
sending more than 10 messages a month, let alone 10 to OUR servers, or
you're a mailing-list driven spam service. There are many
'middle-ground' situations-- social networks, on-demand information
services, mid-range e-commerce sites, and such, which generate a lot
of mail, but it doesn't fit into the usual advertisement
classification. Their rules often don't make sense in these contexts
(what is a legitimate "From" value, for example, if your site
automatically sends a message in response to a user's order, and what
is spoofing? Is it bulk mail when a customer says "Send this RFP to
ten potential suppliers I picked from a list?").

- It's a gateway towards a non-neutral network. Today, it's "certify
you aren't spam to get into our inboxes". Tomorrow it's "Pay us to
get into our inboxes.".

- And... for the privelege, they make no real promises about assuring delivery or keeping you on the whitelist. A handful of malcontent users can defeat the service you provide for hundreds.


Even if the site operator doesn't want to push you into white-list
compliance, the spam filters are often implemented in tragic form:

- Much of the logic is based on restricting mail by IP address. With
today's scarce IP space and higher server efficiency, it's easy to
multiplex 10, 20, 50, or more unrelated domains on the same
mailserver. In the end, you end up with dozens of people punished for
the guilt of one. Hardly a just practice.

- The rules are often vague and non-disclosed. Some warning messages
will point you to a site explaining "your mailing has been throttled
for sending too much mail". What's the threshhold? How long do we
have to wait to resume? The site won't say-- I have to rely on
hearsay of other operators. This seems to be classic "security
through obscurity"-- if you don't explain how you calculate
spamminess, nobody will be able to work around it, eh? It's not them having to explain "Your mail is stuck because (largeISP) is a schmuck, and when they feel willing to grace us with the service their customers paid them to provide again, it will go through."

3. The implementation is a joke.

I could almost tolerate server-side spam blocking if they were direct
about it. Return a failure and explain why in the failure code. That
way, in the event of false positives, the sender can correct matters
and try again in a reasonable time.

Instead, the most common approaches seem to involve deferring
messages. One particular system-- the greylist-- is based entirely on
that premise. They believe that a spammer won't try again on
deferral. I assume it took all of 90 minutes to tweak the mailer
modules of spam-bot software to actually try again.

The deferral solution to spam seems to make no sense. First, it's
based on eventually allowing the mail through. So basically, if my
fake watches were spam initially, if I bash against your door for 16
hours, they become legitimate? The spam is not reduced-- it's just
queued. Maybe the plan is that if they back up the sender's server,
some of the spam will be lost due to a full queue. Second, it
produces a terrible constraint on legitimate email, both in speed and
predictability. I LIKE my email instantaneous-- getting order
acknowledgements within five minutes of purchases, and instant messages
that are really instant. If it's going to take anywhere between four
minutes and three days to deliver, I should just send a letter. Finally,
it tends to make monitoring mail performance and reliability a lot
harder. If a message defers, it should be because the remote server
is acting weird. Instead, we have to figure out if mx.smegheadcorp.com just
dislikes our mail, or is legitimately broken. This is especially fun when you have to pick apart the mail queue to figure out which ONE recipient out of a list of 50+ is deferring. People, do not try to run a full-scale mailing list out of your personal box, but that's another discussion entirely.

Now, I don't have a problem with FILTERING spam on the server. Mark
it as likely spam, but leave it to the user to choose what to do with
it on a per-account basis. But if your idea of solving the spam
problem is to make legitimate email sit on a merry-go-round for 24
hours, I hate to imagine how you solve other problems.

What tops it all off for me is when someone interrupts me when I writing a program or troubleshooting a server and asks the brilliant question "So what's up with all the spam I've been getting lately?" I only have 30-40 users, the filters stop ~3000 messages an hour, and I get interrupted and lose my train of thought because someone is getting 3 or 4 junk mails in their inbox a day.

Then there is the person that tries to send an email that gets blocked at the recipient's end with NDR. When I contact the recipient's IT department they're too stupid to make any sense of it. Even though the problem didn't occur until they upgraded their mail system, they claim it's on my end, ignoring what the NDR says.

The other thing I don't get is how the users put up with it on the recipient side.

Slow and erratic email service is the sort of thing you'd expect them to raise hell over. Are they so terrified of a little spam, they'll put up with it?

I honestly find spam more comical than terrible-- that people are expected to buy things spelled so badly, or trust some dead Nigerian...

Email is effing horrible.

My buddy works for Internet Security Services [now part of IBM]. Their Lotus Notes implementation sucks so bad that everyone complains about it.

Let's make sure we understand:
1} IBM is the world's oldest, most respected computer company.
2] They make Lotus Notes
3] They chose Lotus Notes for their corporate network; they could buy any program they wanted.
4] As they make Lotus Notes, they have access to the source code.
5] As they make Lotus Notes they have access to the developers of Lotus Notes, guys who know more about email than I will ever know about everything.
6] It's still a slow POS with security issues that prevent their own programmers from working.

It boggles the mind.

-MF