Apex Community Forums - View Single Post

Hak Foo · 21-March-08, 11:53 PM

Warning: Long Diatribe is Long.

Although my nominal profession is web developer, I end up doing most
of the technical service. If the server needs reconfiguration, I do
it. If someone's PC catches fire, I have to smother it with blankets.
I write the automation scrips to backup the server. Fair enough,
it's a small company and I was the first person there who could tell
the difference between 'cp' and 'rm'.

The one true burr under my saddle is having to deal with email. It
isn't so bad if people are screaming in the event of an outage, and
it's only mildly annoying when they forget their passwords. But what
drives me most insane is deliverability issues. I hate staring tensely at mail queues, surfing through maillog, and poking qmail to retry all those queued messages in a reasonable timeframe. It's clear: ISPs have
no business doing server-side blocking of junk mail.

1. They don't know what spam is.

As good as filtering technology has become, no system is absolutely
correct. This is both because filters don't understand content, and
because they don't understand users or their intended uses. A
legitimate bank statement looks remarkably like a quality phishing
job. That perfectly formatted gibberish sent to ten addresses? It
was a $1200 e-commerce order being forwarded to appropriate
departments for fulfillment. And sadly, for some people, the
mis-spelled Viagra solicitation from a site in Hong Kong is
desperately awaited.

It's further complicated by the fact even users don't know what spam
is. I'm sure you-- or someone you know-- has simply used the "file as
spam" button to make a once-requested newsletter go away. The whole
"never, ever click a 'remove' link" indoctrination hardly helps that
particular problem. Who gets penalized? The list issuer, who may
well have gone to great lengths for compliance.

Furthermore, there are often legitimate times to pass spam. Many
users, for example, appeciate a "branded" email address, but want it
to all be forwarded to their ISP address. It would be impolite, if
not downright negligent, to try to go through and delete what we think
is spam before it was passed on.

2. These filters tend to set their own rules. I note several of the
large mail providers-- unsurprisingly the ones with the most obnoxious
filtering problems-- seem to be all to happy to get you to sign up for
white-list service. This is an disaster in many distinct ways:

- They frequently demand behaviour well beyond compliance with anti-spam laws.

- They tend to be organized only in the concept of two classes of
users. You're either a small corporation who never has any business
sending more than 10 messages a month, let alone 10 to OUR servers, or
you're a mailing-list driven spam service. There are many
'middle-ground' situations-- social networks, on-demand information
services, mid-range e-commerce sites, and such, which generate a lot
of mail, but it doesn't fit into the usual advertisement
classification. Their rules often don't make sense in these contexts
(what is a legitimate "From" value, for example, if your site
automatically sends a message in response to a user's order, and what
is spoofing? Is it bulk mail when a customer says "Send this RFP to
ten potential suppliers I picked from a list?").

- It's a gateway towards a non-neutral network. Today, it's "certify
you aren't spam to get into our inboxes". Tomorrow it's "Pay us to
get into our inboxes.".

- And... for the privelege, they make no real promises about assuring delivery or keeping you on the whitelist. A handful of malcontent users can defeat the service you provide for hundreds.

Even if the site operator doesn't want to push you into white-list
compliance, the spam filters are often implemented in tragic form:

- Much of the logic is based on restricting mail by IP address. With
today's scarce IP space and higher server efficiency, it's easy to
multiplex 10, 20, 50, or more unrelated domains on the same
mailserver. In the end, you end up with dozens of people punished for
the guilt of one. Hardly a just practice.

- The rules are often vague and non-disclosed. Some warning messages
will point you to a site explaining "your mailing has been throttled
for sending too much mail". What's the threshhold? How long do we
have to wait to resume? The site won't say-- I have to rely on
hearsay of other operators. This seems to be classic "security
through obscurity"-- if you don't explain how you calculate
spamminess, nobody will be able to work around it, eh? It's not them having to explain "Your mail is stuck because (largeISP) is a schmuck, and when they feel willing to grace us with the service their customers paid them to provide again, it will go through."

3. The implementation is a joke.

I could almost tolerate server-side spam blocking if they were direct
about it. Return a failure and explain why in the failure code. That
way, in the event of false positives, the sender can correct matters
and try again in a reasonable time.

Instead, the most common approaches seem to involve deferring
messages. One particular system-- the greylist-- is based entirely on
that premise. They believe that a spammer won't try again on
deferral. I assume it took all of 90 minutes to tweak the mailer
modules of spam-bot software to actually try again.

The deferral solution to spam seems to make no sense. First, it's
based on eventually allowing the mail through. So basically, if my
fake watches were spam initially, if I bash against your door for 16
hours, they become legitimate? The spam is not reduced-- it's just
queued. Maybe the plan is that if they back up the sender's server,
some of the spam will be lost due to a full queue. Second, it
produces a terrible constraint on legitimate email, both in speed and
predictability. I LIKE my email instantaneous-- getting order
acknowledgements within five minutes of purchases, and instant messages
that are really instant. If it's going to take anywhere between four
minutes and three days to deliver, I should just send a letter. Finally,
it tends to make monitoring mail performance and reliability a lot
harder. If a message defers, it should be because the remote server
is acting weird. Instead, we have to figure out if mx.smegheadcorp.com just
dislikes our mail, or is legitimately broken. This is especially fun when you have to pick apart the mail queue to figure out which ONE recipient out of a list of 50+ is deferring. People, do not try to run a full-scale mailing list out of your personal box, but that's another discussion entirely.

Now, I don't have a problem with FILTERING spam on the server. Mark
it as likely spam, but leave it to the user to choose what to do with
it on a per-account basis. But if your idea of solving the spam
problem is to make legitimate email sit on a merry-go-round for 24
hours, I hate to imagine how you solve other problems.

21-March-08, 11:53 PM		#1 (permalink)
Hak Foo Apex Elite Tech	Sysadmin Rant Warning: Long Diatribe is Long. Although my nominal profession is web developer, I end up doing most of the technical service. If the server needs reconfiguration, I do it. If someone's PC catches fire, I have to smother it with blankets. I write the automation scrips to backup the server. Fair enough, it's a small company and I was the first person there who could tell the difference between 'cp' and 'rm'. The one true burr under my saddle is having to deal with email. It isn't so bad if people are screaming in the event of an outage, and it's only mildly annoying when they forget their passwords. But what drives me most insane is deliverability issues. I hate staring tensely at mail queues, surfing through maillog, and poking qmail to retry all those queued messages in a reasonable timeframe. It's clear: ISPs have no business doing server-side blocking of junk mail. 1. They don't know what spam is. As good as filtering technology has become, no system is absolutely correct. This is both because filters don't understand content, and because they don't understand users or their intended uses. A legitimate bank statement looks remarkably like a quality phishing job. That perfectly formatted gibberish sent to ten addresses? It was a $1200 e-commerce order being forwarded to appropriate departments for fulfillment. And sadly, for some people, the mis-spelled Viagra solicitation from a site in Hong Kong is desperately awaited. It's further complicated by the fact even users don't know what spam is. I'm sure you-- or someone you know-- has simply used the "file as spam" button to make a once-requested newsletter go away. The whole "never, ever click a 'remove' link" indoctrination hardly helps that particular problem. Who gets penalized? The list issuer, who may well have gone to great lengths for compliance. Furthermore, there are often legitimate times to pass spam. Many users, for example, appeciate a "branded" email address, but want it to all be forwarded to their ISP address. It would be impolite, if not downright negligent, to try to go through and delete what we think is spam before it was passed on. 2. These filters tend to set their own rules. I note several of the large mail providers-- unsurprisingly the ones with the most obnoxious filtering problems-- seem to be all to happy to get you to sign up for white-list service. This is an disaster in many distinct ways: - They frequently demand behaviour well beyond compliance with anti-spam laws. - They tend to be organized only in the concept of two classes of users. You're either a small corporation who never has any business sending more than 10 messages a month, let alone 10 to OUR servers, or you're a mailing-list driven spam service. There are many 'middle-ground' situations-- social networks, on-demand information services, mid-range e-commerce sites, and such, which generate a lot of mail, but it doesn't fit into the usual advertisement classification. Their rules often don't make sense in these contexts (what is a legitimate "From" value, for example, if your site automatically sends a message in response to a user's order, and what is spoofing? Is it bulk mail when a customer says "Send this RFP to ten potential suppliers I picked from a list?"). - It's a gateway towards a non-neutral network. Today, it's "certify you aren't spam to get into our inboxes". Tomorrow it's "Pay us to get into our inboxes.". - And... for the privelege, they make no real promises about assuring delivery or keeping you on the whitelist. A handful of malcontent users can defeat the service you provide for hundreds. Even if the site operator doesn't want to push you into white-list compliance, the spam filters are often implemented in tragic form: - Much of the logic is based on restricting mail by IP address. With today's scarce IP space and higher server efficiency, it's easy to multiplex 10, 20, 50, or more unrelated domains on the same mailserver. In the end, you end up with dozens of people punished for the guilt of one. Hardly a just practice. - The rules are often vague and non-disclosed. Some warning messages will point you to a site explaining "your mailing has been throttled for sending too much mail". What's the threshhold? How long do we have to wait to resume? The site won't say-- I have to rely on hearsay of other operators. This seems to be classic "security through obscurity"-- if you don't explain how you calculate spamminess, nobody will be able to work around it, eh? It's not them having to explain "Your mail is stuck because (largeISP) is a schmuck, and when they feel willing to grace us with the service their customers paid them to provide again, it will go through." 3. The implementation is a joke. I could almost tolerate server-side spam blocking if they were direct about it. Return a failure and explain why in the failure code. That way, in the event of false positives, the sender can correct matters and try again in a reasonable time. Instead, the most common approaches seem to involve deferring messages. One particular system-- the greylist-- is based entirely on that premise. They believe that a spammer won't try again on deferral. I assume it took all of 90 minutes to tweak the mailer modules of spam-bot software to actually try again. The deferral solution to spam seems to make no sense. First, it's based on eventually allowing the mail through. So basically, if my fake watches were spam initially, if I bash against your door for 16 hours, they become legitimate? The spam is not reduced-- it's just queued. Maybe the plan is that if they back up the sender's server, some of the spam will be lost due to a full queue. Second, it produces a terrible constraint on legitimate email, both in speed and predictability. I LIKE my email instantaneous-- getting order acknowledgements within five minutes of purchases, and instant messages that are really instant. If it's going to take anywhere between four minutes and three days to deliver, I should just send a letter. Finally, it tends to make monitoring mail performance and reliability a lot harder. If a message defers, it should be because the remote server is acting weird. Instead, we have to figure out if mx.smegheadcorp.com just dislikes our mail, or is legitimately broken. This is especially fun when you have to pick apart the mail queue to figure out which ONE recipient out of a list of 50+ is deferring. People, do not try to run a full-scale mailing list out of your personal box, but that's another discussion entirely. Now, I don't have a problem with FILTERING spam on the server. Mark it as likely spam, but leave it to the user to choose what to do with it on a per-account basis. But if your idea of solving the spam problem is to make legitimate email sit on a merry-go-round for 24 hours, I hate to imagine how you solve other problems.