• Apple’s new low: censoring a dictionary #
• Followup on the Apple dictionary story. #

Also, the webcomic Cat and Girl was Sent from my iPhone (via @brionv) #

Eventually I got too busy to read them, and the back-issues piled up unread, and I decided to let my subscription lapse. But earlier this year, I decided to re-up with the shorter, free version, and it’s still as good as ever.

This week’s issue included a disappointing story: even though they practice — in fact, probably helped originate — responsible list management, Yahoo is blocking them as spammers. Why? Because people are signing up for the list, then deciding they don’t want it anymore, and instead of unsubscribing, hitting the “Report as Spam” button. Yahoo has apparently taken those spam reports at face value, and blocked everyone’s copy of the newsletter.

Clearly, some people are unclear on what “spam” means. It’s not just “mail I don’t want.” It’s “mass mail I don’t want and didn’t ask for.”

That, and I’m sure some people don’t realize that their reports are being used to train everyone’s filters. I remember a co-worker explaining a few years ago that he’d trained Gmail to send the SourceForge newsletters (or something similar) straight into his spam folder. I commented that they might be using that data to train their sitewide filters, and he said something like, “I hope not.”

Using user feedback to train sitewide or network-wide (such as Cloudmark, or Akismet) filters is a powerful technique. Some people will catch the leading edge of a spam attack, and that data can be used to protect others as the attack continues. Some will check their mail sooner, and that data can be used to re-filter messages that have been received, but not yet viewed.

Unfortunately, it also can give a lot of power to people who are either unclear on the criteria being used or have an axe to grind, unless you include measures to (a) contain the impact or (b) keep track of each reporter’s reliability. I know Cloudmark factors in the reporter’s reputation, for instance. And I suspect that AOL does, at least in some cases, limit measures such as blocking to specific recipients, but I can’t be certain.

Anyway, to summarize:

• Use the Report Spam button responsibly.  If you actually subscribed to it, it isn’t spam unless they refuse to remove you from the list.
• Check out This is True.  You may laugh, you may groan, you may think, or you may get pissed off at the world — or all of the above.  It’s certainly worth a look.

(I really should have finished writing this yesterday, before someone submitted the original story to Slashdot. Posting about it to get the word out seems kind of redundant now. Heck, now that I think about it, I should have submitted the original to Slashdot. Oh, well.

On the tech side, it’s similar to SpamAssassin’s whitelist_from_spf and whitelist_from_dkim features. Both allow you to specify a sender to whitelist, and it will only give a message special treatment if it can verify the sender.

On the user-interface side, it’s similar to EC certificates, in that it tries to highlight a “good” class of messages rather than flag or filter out a “bad” class.

It’s not a bad idea, actually, and now that I’m surprised I haven’t seen something similar in other email clients. It’s sort of like setting up custom rings or images for images on your cell phone address book

They seem to be focused on webmail and Outlook so far, and only on Windows, but it looks like the perfect candidate for a Thunderbird extension. They do have a sign-up form to notify you when they add support for various programs and OSes, and I was pleased to see not only Thunderbird and Mac OS listed, but Linux as well. Too often, Linux gets forgotten in the shuffle to ensure compatibility with every Windows variation.

HIGHLY CONFIDENTIAL REQUESTING

What made this funny (aside from the bad grammar) was the fact that the To: line contained over 1,200 addresses!

Ah, this is obviously some strange use of the word confidential that I wasn’t previously aware of!

Pick one domain name for your business. Just one. Don’t use any other domains in your emails, even if you want to keep order confirmations separate from promotions. If you contract out for some other company to send out a newsletter or survey to your customers, insist that they send it out using your own domain name. If you’re using DomainKeys or SPF, make sure they’re authorized or send it yourself. And don’t even think of making the links through redirection scripts, even if you really want to track which subscribers are clicking.

Why?

Two words: Spam and fraud.

We, as email admins, want to separate the wheat from the chaff among the mail coming into our organization. Why, why do you insist on making your mailings look like chaff?

Banks—you know how rampant phishing is. You can make it easy for your customers to know whether a message came from you or from a fraud ring. If it comes from yourbank.com, and all the links are to yourbank.com, it’s legit. If it comes from anything else—even yourbankonline.com—it’s suspect. But when you can’t decide between citi.com, citibank.com, citicards.com, citicorp.com, and citibankcards.com, how are we to know that when some phisherman sets up citibankcardsonline.com, it’s not you?

And when you contract out to some third-party promo list and it comes from m0.net, and the login links redirect through them instead of going straight to you, what the hell are we supposed to think? How are we supposed to know that yes, this really did come from you and not some scam artist in Uzbekistan?

And those of you who insist on doing all the cutesy graphical tricks with HTML mail. If we know about you, we can whitelist you. But it helps if you don’t make yourselves moving targets! Yes, Deep Discount DVD, I’m talking to you. I have you whitelisted at deepdiscountdvd.com, so why on Earth would you take the risk of sending me mail as deepdiscountdvdpromotions.com? And why, why was this morning’s “Your order had shipped” message from DeepDiscountDVD[at]mail.infinityresources.com??? (OK, I figure that last bit was probably just misconfigured, and it was plain text, but still…) I know you have to keep your costs down, but you could at least hire a network consultant to make sure your mail servers are set up correctly!

There’s only one problem. Since SpamAssassin and ClamAV do such a good job of catching the phishing scams before they reach my inbox, Thunderbird has yet to catch any actual phish. But there’ve been a lot of false positives. It’s hit LiveJournal reply notices, newsletters from IEEE and Golden Key, a Spam Karma notice from my own blog, and I’ve seen it on both outbid notices and updates to saved searches from eBay.

I found myself wondering just how Thunderbird’s phishing detection decides that a message is suspicious—and how to teach it that the next LJ notice isn’t a scam.

The Thunderbird support website doesn’t seem to have been updated yet. Most of the articles I’ve found only talk about TB adding the feature, not how it works. The best information I found was this Mozillazine forum thread, which included a link to the actual code that makes the decision, in phishingDetector.js. Thunderbird looks at the following:

• Links that only use an IP address, including dotted decimal, octal, hex, dword, or some mixed encoding.
• Links that claim to go to one site, but actually go to another. (Phishers do this to fool you into going to their site. Legit mailing lists sometimes do this with redirectors for tracking purposes.)
• Forms embedded in the email. (This explains the LiveJournal notices.)

It also appears to trap text URLs containing HTML-escaped characters, which explains the Spam Karma reports. In this case the report includes a spammer’s link with ​ in the hostname. The message is plain text, so Thunderbird leaves the entity as-is when displaying it…but decodes it when it creates the link. Result: a link where the text and URL don’t match.

The easiest way to prevent it from freaking out over the next message? Add the sender to your address book. I’m not sure that’s a great idea, since a phisher could guess which addresses you have saved and spoof them, but it’s at least simple. I guess I’ll find out whether it works the next time I get a reply notice from LJ. Update: Adding the sender to your address book doesn’t seem to have any effect.

Update 2 (July 12, 2006): The comment thread’s gotten long enough that I can see people might miss this, so here’s how to disable it:

1. Open Options or Preferences (this will be under the Tools menu on Windows, Thunderbird on Mac, or Edit on Linux).
2. Click on Privacy (there should be a big padlock icon).
3. Click on the E-mail Scams tab.
4. Disable the “Check mail messages for email scams” option and click on Close.

That’s it.

Either way, the new comment notice went out, and the recipient sent me a spam complaint. I apologized and removed him from the update list, but it moves “accidental spam” from a theoretical risk to an observed problem. I’ve disabled the subscription plugin until I have a chance to figure this out.

The good news is that Subscribe to Comments 2.0 is out now, so I should be able to upgrade when I get a chance. The bad news is that it doesn’t seem to have added a confirmation step, meaning it’s still (effectively) opt-out. Sure, you have to opt-in to get it in the first place…but the fact is that anyone can opt you in just by giving your email address instead of their own.

Enter the dictionary attacks. An awful lot of those standard accounts are three-letter names—rpm, gdm, bin, adm, etc. Spammers trying to guess addresses made up of three initials landed on these addresses, confirmed them, and added them to their lists. The system accounts began collecting spam.

Eventually we locked things down so that only “real” accounts would accept mail from outside. But here was this steady stream of 100% spam we could use to help train our filters.

The funny thing: these days, nearly all of it is for sex-related drugs or body part enlargements. Sent to software!

(Incidentally, if you can read this sentence, don’t send mail to ramblo@hyperborea.org.)

1. Spammers send mail directly to victims.
2. Server admins block by source, victims complain and try to get spammers kicked off their networks.
3. Spammers relay through third-party servers to disguise their origin.
4. Server admins shut close relays, and block mail from open relays.
5. Spammers relay through trojaned zombies straight to victims.
6. Network admins block outgoing mail traffic except through their servers.
7. Spammers relay through zombies’ ISPs’ mail servers.
8. ????

We’re in the early stages of step 6, with broadband ISPs starting to block outgoing direct-to-MX mail traffic. The obvious response by spammers is, of course, to get their virus-writing partners to add code that extracts settings from the infected system’s mail program, and send through the ISP just like the actual user would.

At this point the problem changes. To use a car metaphor, first spammers drove their own cars, then they stole trucks, and now they’re stealing your car while you’re at work and driving it off-road. Soon they’ll be stealing your car, but keeping to city streets and using a fake drivers’ license with your name on it. So blocking by source and authentication won’t be enough.

The next step will probably be dynamic blocks on outgoing mail based on some sort of traffic analysis. This would be things like temporarily blocking mail from client IPs that send out viruses, and notifying the customer. Perhaps using statistical analysis like credit card fraud protection. (Hmm, this customer normally sends 10-15 emails a day, but seems to have sent 1000 in the past hour.)

We may be reaching the limits of blocking by source—or at least blocking by immediate source. If some sort of sender verification (SPF or DomainKeys) really takes off, it may be possible to extend it further.