Jan 15 2005

mt-proxyplug shut down comments through proxies

A few days ago I posted Deny Comment Spam from open proxies in MovableType which showed a technique to limit comment submissions through proxies. Unfortunately there are a number of issues with the small plugin, which made me create mt-proxyplug, presented in this article.

How it all started

I am a longtime user of Jay Allen’s Blacklist and was happy with it for a long time. Recently I could not keep up with adding new keywords/urls to the black list. There seem to be a million variations of “Texas Hold’em” out there and I ended up adding those items slowly to the Blacklist system (in the end I actually added “texas” as a Blacklist item only to find a “texa$ H0ld’em” the next day in my list of moderated comments).

I started to look more carefully at the offending posts and investigated the submitting IP addresses in detail. Soon I realized that most of the stuff was coming from public proxy servers. Companies stupid enough to run public proxies and hijacked user systems are on top of the list of systems that submitted spam to my server.

Monitoring the proxies

My apache configuration was changed to include some proxy specific information in my access_logs. I changed the line:
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined
to
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{HTTP_X_FORWARDED_FOR}e" combined

This means that the apache server will also log the contents of the environment variable “HTTP_X_FORWARDED_FOR” to the access_log, whenever that environment variable is present. And the environment variable is present if the current request contains a “X-Forwarded-For:” header item. The presence of this item is almost always a clear indication that the request was handled by a proxy server. Items that would have previously been logged like this:

200.242.249.70 - - [15/Jan/2005:11:31:52 -0700] "POST /blog/mt-comments.cgi HTTP/1.0" 302 0 "http://www.kahunaburger.com/blog/mt-comments.cgi?entry_id=113" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322)"

suddenly looked like this:

200.242.249.70 - - [15/Jan/2005:11:31:52 -0700] "POST /blog/mt-comments.cgi HTTP/1.0" 302 0 "http://www.kahunaburger.com/blog/mt-comments.cgi?entry_id=113" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322)" 168.41.192.0

(note the addition of the IP address at the end of the line)

That’s when I started focusing on proxies and developed mt-commentproxyblock, which evolved into mt-proxyplug.

What does it do?

mt-proxyplug when installed on a MovableType system will act as a ‘CommentFilter’. Every comment submission is passed through it, before it is committed to the database. The plugin will inspect the remote IP address of the system that submits the comment. First it will check if there is a “X-Forwarded-For:” header item in the current comment submission request. The mere presence of the header field is an indication that stuff is being submitted through a proxy server. It will then query the Distributed Sender Blackhole List and the Blitzed Open Proxy Monitor List for entries for the submitting remote IP address. If either one knows about the IP address, then we assume that the comment comes from a known public proxy system.

In a last test the remote system is probed on a number of common proxy ports. We try to get access through the system to a well-known and relatively stable host on the internet. If this request is processed successfully on any of the probed ports we also assume that the remote end is indeed a public proxy.

If any one of the above tests is positive we are not executing the other tests and simply flag the current comment submission as suspicious.

A configuration section at the top of the file allows for customization of the list of tests the plugin should run.
There is also a CACHE_COUNT definition that specifies how many found proxies the plugin should keep track of (this will make it much quicker on subsequent requests, if a proxy is used numerous times in a row).

A log of the plugin’s actions is also provided in MT’s Activity Log. Here’s just a small section from my current log:

Since I installed the plugin on kahunaburger.com’s blog it has caught 121 of 122 comment submissions. The one that slipped through was actually caught by mt-spamassassin. During the same time period I also received 4 good comment submissions which made it through the system without any problems.

How to use mt-proxyplug

Just drop the file below into your MovableType’s plugins folder. Modify the “settings” section to your liking (the default values are the recommended values) and you’re set.
No other modules are required (I assume that IO::Socket is available on all newer perl installations). HTTP::CheckProxy (as used in the previous version of the plugin) has been dropped, because it would report false positives (or is it “true negatives”?).

Update 01/20/2005: I’ve updated the plugin below to version 0.6. Two changes since the original version:
1) I set CHECK_LIST_DSBL_ORG to “0” by default, based on ioerror.us’s comments here
2) I fixed the require list at the top of the file to include “LWP::UserAgent” after receiving a problem report from Chris.

You can download the plugin here: mt-proxyplug.pl.gz (2.5KB, gzip)

10 Responses to “mt-proxyplug shut down comments through proxies”

  • stan Says:

    I was wondering if this Plugin will only work with MT version 3.x or can we instal on older versions too? I followed your brief directions above, but unfortunately got lost since I can not see that screen shot you have posted.

  • IO ERROR Says:

    I recommend putting HTTP_FORWARDED_FOR before UserAgent, so that you can still use standard tools like grep, cut, etc., to break apart and analyze the logs. This might not be important to you, but your web analysis tools might need it.

  • IO ERROR Says:

    On further consideration I’d replace %l (the second field) with HTTP_FORWARDED_FOR. This field is not used so it makes a much better candidate, and as was pointed out to me, shifting the User-Agent field over might confuse some web analysis tools.

  • Jamie McCarthy Says:

    Do either of the open proxy lists ever catch anything? Since those are specifically for SMTP proxies I would be surprised if very many of those machines are also HTTP proxies, and I’d figure it wasn’t worth the effort to check them.

  • Justin Mason Says:

    Hey — as one of the SA developers, I like what you’re doing here, very interesting stuff! I never would have thought SA could make much of an inroad into comment spam, given how different HTTP and SMTP are.

    Jamie’s point about the open proxy lists is a good one — many of those machines are indeed running HTTP proxies that are being misused (using “CONNECT”) as SMTP proxies, so you may find an overlap there. SA will do that check for you if you’re running without the -L switch on spamd, and if you generate plausible-looking Received header in the “mail” you pass to it.

  • Chris Collins Says:

    I have been getting the following error with this plugin. I wonder what I should be checking to see why. The webhost I use uses Perl 5.8.4.

    {
    Proxy Plug for Movable Type v0.5 died with: Can’t locate object method “new” via package “LWP::UserAgent” at plugins/mt-proxyplug.pl line 201.
    }

  • Gabe Misura Says:

    Does this work with MT 2.611? I put the file in my plugin directory and seems to be doing nothing. I’ve turned on LOG_DEBUG ( => 1) and still i don’t see anything in the mt_log mysql table.

  • Mark Odell Says:

    Since it seems that dsbl.org is not broad enough, and since sbl-xbl.spamhaus.org includes Blitzed OPM, is there any chance of allowing the admin to select different/additional/broader DNSBLs, even (dare I say it? ๐Ÿ™‚ SPEWS?

  • Paul Says:

    Hello all ๐Ÿ™‚