mt-proxyplug shut down comments through proxies
A few days ago I posted Deny Comment Spam from open proxies in MovableType which showed a technique to limit comment submissions through proxies. Unfortunately there are a number of issues with the small plugin, which made me create mt-proxyplug, presented in this article.
How it all started
I am a longtime user of Jay Allen’s Blacklist and was happy with it for a long time. Recently I could not keep up with adding new keywords/urls to the black list. There seem to be a million variations of “Texas Hold’em” out there and I ended up adding those items slowly to the Blacklist system (in the end I actually added “texas” as a Blacklist item only to find a “texa$ H0ld’em” the next day in my list of moderated comments).
I started to look more carefully at the offending posts and investigated the submitting IP addresses in detail. Soon I realized that most of the stuff was coming from public proxy servers. Companies stupid enough to run public proxies and hijacked user systems are on top of the list of systems that submitted spam to my server.
Monitoring the proxies
My apache configuration was changed to include some proxy specific information in my access_logs. I changed the line:
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined
to
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{HTTP_X_FORWARDED_FOR}e" combined
This means that the apache server will also log the contents of the environment variable “HTTP_X_FORWARDED_FOR” to the access_log, whenever that environment variable is present. And the environment variable is present if the current request contains a “X-Forwarded-For:” header item. The presence of this item is almost always a clear indication that the request was handled by a proxy server. Items that would have previously been logged like this:
200.242.249.70 - - [15/Jan/2005:11:31:52 -0700] "POST /blog/mt-comments.cgi HTTP/1.0" 302 0 "http://www.kahunaburger.com/blog/mt-comments.cgi?entry_id=113" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322)"
suddenly looked like this:
200.242.249.70 - - [15/Jan/2005:11:31:52 -0700] "POST /blog/mt-comments.cgi HTTP/1.0" 302 0 "http://www.kahunaburger.com/blog/mt-comments.cgi?entry_id=113" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322)" 168.41.192.0
(note the addition of the IP address at the end of the line)
That’s when I started focusing on proxies and developed mt-commentproxyblock, which evolved into mt-proxyplug.
What does it do?
mt-proxyplug when installed on a MovableType system will act as a ‘CommentFilter’. Every comment submission is passed through it, before it is committed to the database. The plugin will inspect the remote IP address of the system that submits the comment. First it will check if there is a “X-Forwarded-For:” header item in the current comment submission request. The mere presence of the header field is an indication that stuff is being submitted through a proxy server. It will then query the Distributed Sender Blackhole List and the Blitzed Open Proxy Monitor List for entries for the submitting remote IP address. If either one knows about the IP address, then we assume that the comment comes from a known public proxy system.
In a last test the remote system is probed on a number of common proxy ports. We try to get access through the system to a well-known and relatively stable host on the internet. If this request is processed successfully on any of the probed ports we also assume that the remote end is indeed a public proxy.
If any one of the above tests is positive we are not executing the other tests and simply flag the current comment submission as suspicious.
A configuration section at the top of the file allows for customization of the list of tests the plugin should run.
There is also a CACHE_COUNT definition that specifies how many found proxies the plugin should keep track of (this will make it much quicker on subsequent requests, if a proxy is used numerous times in a row).
A log of the plugin’s actions is also provided in MT’s Activity Log. Here’s just a small section from my current log:
Since I installed the plugin on kahunaburger.com’s blog it has caught 121 of 122 comment submissions. The one that slipped through was actually caught by mt-spamassassin. During the same time period I also received 4 good comment submissions which made it through the system without any problems.
How to use mt-proxyplug
Just drop the file below into your MovableType’s plugins folder. Modify the “settings” section to your liking (the default values are the recommended values) and you’re set.
No other modules are required (I assume that IO::Socket is available on all newer perl installations). HTTP::CheckProxy (as used in the previous version of the plugin) has been dropped, because it would report false positives (or is it “true negatives”?).
Update 01/20/2005: I’ve updated the plugin below to version 0.6. Two changes since the original version:
1) I set CHECK_LIST_DSBL_ORG to “0″ by default, based on ioerror.us’s comments here
2) I fixed the require list at the top of the file to include “LWP::UserAgent” after receiving a problem report from Chris.
You can download the plugin here: mt-proxyplug.pl.gz (2.5KB, gzip)
Comments(10)
return ACCEPT_RESPONSE;
}
$toSend=substr($toSend,$written);
}
# close writing end of socket
$socket->shutdown(1);
# suck in response from SpamAssassin
my $response;
while(1) {
my $buffer;
unless(defined($socket->recv($buffer, 1024))) {
return ACCEPT_RESPONSE;
}
last unless(length($buffer));
$response .= $buffer;
}
# trim whitespace off the beginning of the response
$response =~ s/^\s*//;
# check if it is really a SpamAssassin response
return ACCEPT_RESPONSE unless ($response =~ /^spamd\/[\d\.]+/i);
# now find "Spam: True|False ; score / limit" header
return ACCEPT_RESPONSE
unless ($response =~ /spam:\s*(\S+)\s*;\s*([\d\.]+)\s*\/\s*([\d\.]+)/is);
my($flag,$score,$limit)=($1,$2,$3);
#if($flag =~ /false/i) {
#print STDERR "[".scalar(localtime())."] no spam:\n$message\n";
#}
print STDERR "[".scalar(localtime())."] spam $flag from " .
join("/",$comment->author,$comment->email,$comment->ip) .
": score $score (limit $limit)\n";
if($flag =~ /false/i) {
if($score > $mt_moderate) {
print STDERR "[".scalar(localtime())."] moderating comment\n";
$comment->visible(0);
}
return ACCEPT_RESPONSE;
}
# log a line to the error_log
return DENY_RESPONSE;
}
# rfc822_date
#
# generate a GMT date according to rfc822
sub rfc822_date {
# offset in hours (from Mail::Sendmail)
my $offset = sprintf "%.1f", (timegm(localtime) - time) / 3600;
my $minutes = sprintf "%02d", abs( $offset - int($offset) ) * 60;
my $TZ = sprintf("%+03d", int($offset)) . $minutes;
return POSIX::strftime("%a, %d %b %Y %T $TZ",localtime(time()));
}
1;