Kahunaburger.com is proud to present the first “Referer Spammer of the Month Award”.
It was a close race between the two favorites and only a few hundred referer spam entries separate them, but in the end, there can only be one winner.
(more…)
I just banned webclipping.com’s host from access to my website. They’ve been rather rude in the way they suck down content and make money from it. webclipping.com, if you’re using my content, do it in a nice way and fix your spider, so that it does not download the same content over and over again.
People with similar experience can be found at http://troyandgay.com/index.php?p=2906.
Access.log evidence in the extended entry.
(more…)
I thought this was interested.
On 5/5/2002 I photographed Candice and Frisco on one of our walks with the new puppy:
On 1/16/2005 I did the same, but this time with a different puppy:
There is actually a frigthening development between those pictures (it’s even more apparent in the color version): All those pinion trees in the background on the left side are dead in the second picture. This is a direct result of the bark beetle attacks in the Santa Fe area.
Pias normal day includes a nap at around 9am in the morning. Sometimes she falls asleep within minutes and sometimes it takes a bit longer. Here’s an account of a 30 min delay from the time she was put down until she finally falls asleep. The motion detection software for the wireless webcam (see Perl bringing Pia to the Web) captured numerous frame while she was playing in her crib. I combined all of them into a short video:
A few days ago I posted Deny Comment Spam from open proxies in MovableType which showed a technique to limit comment submissions through proxies. Unfortunately there are a number of issues with the small plugin, which made me create mt-proxyplug, presented in this article.
How it all started
I am a longtime user of Jay Allen’s Blacklist and was happy with it for a long time. Recently I could not keep up with adding new keywords/urls to the black list. There seem to be a million variations of “Texas Hold’em” out there and I ended up adding those items slowly to the Blacklist system (in the end I actually added “texas” as a Blacklist item only to find a “texa$ H0ld’em” the next day in my list of moderated comments).
I started to look more carefully at the offending posts and investigated the submitting IP addresses in detail. Soon I realized that most of the stuff was coming from public proxy servers. Companies stupid enough to run public proxies and hijacked user systems are on top of the list of systems that submitted spam to my server.
Monitoring the proxies
My apache configuration was changed to include some proxy specific information in my access_logs. I changed the line:
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined
to
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{HTTP_X_FORWARDED_FOR}e" combined
This means that the apache server will also log the contents of the environment variable “HTTP_X_FORWARDED_FOR” to the access_log, whenever that environment variable is present. And the environment variable is present if the current request contains a “X-Forwarded-For:” header item. The presence of this item is almost always a clear indication that the request was handled by a proxy server. Items that would have previously been logged like this:
200.242.249.70 - - [15/Jan/2005:11:31:52 -0700] "POST /blog/mt-comments.cgi HTTP/1.0" 302 0 "http://www.kahunaburger.com/blog/mt-comments.cgi?entry_id=113" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322)"
suddenly looked like this:
200.242.249.70 - - [15/Jan/2005:11:31:52 -0700] "POST /blog/mt-comments.cgi HTTP/1.0" 302 0 "http://www.kahunaburger.com/blog/mt-comments.cgi?entry_id=113" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322)" 168.41.192.0
(note the addition of the IP address at the end of the line)
That’s when I started focusing on proxies and developed mt-commentproxyblock, which evolved into mt-proxyplug.
What does it do?
mt-proxyplug when installed on a MovableType system will act as a ‘CommentFilter’. Every comment submission is passed through it, before it is committed to the database. The plugin will inspect the remote IP address of the system that submits the comment. First it will check if there is a “X-Forwarded-For:” header item in the current comment submission request. The mere presence of the header field is an indication that stuff is being submitted through a proxy server. It will then query the Distributed Sender Blackhole List and the Blitzed Open Proxy Monitor List for entries for the submitting remote IP address. If either one knows about the IP address, then we assume that the comment comes from a known public proxy system.
In a last test the remote system is probed on a number of common proxy ports. We try to get access through the system to a well-known and relatively stable host on the internet. If this request is processed successfully on any of the probed ports we also assume that the remote end is indeed a public proxy.
If any one of the above tests is positive we are not executing the other tests and simply flag the current comment submission as suspicious.
A configuration section at the top of the file allows for customization of the list of tests the plugin should run.
There is also a CACHE_COUNT definition that specifies how many found proxies the plugin should keep track of (this will make it much quicker on subsequent requests, if a proxy is used numerous times in a row).
A log of the plugin’s actions is also provided in MT’s Activity Log. Here’s just a small section from my current log:
Since I installed the plugin on kahunaburger.com’s blog it has caught 121 of 122 comment submissions. The one that slipped through was actually caught by mt-spamassassin. During the same time period I also received 4 good comment submissions which made it through the system without any problems.
How to use mt-proxyplug
Just drop the file below into your MovableType’s plugins folder. Modify the “settings” section to your liking (the default values are the recommended values) and you’re set.
No other modules are required (I assume that IO::Socket is available on all newer perl installations). HTTP::CheckProxy (as used in the previous version of the plugin) has been dropped, because it would report false positives (or is it “true negatives”?).
Update 01/20/2005: I’ve updated the plugin below to version 0.6. Two changes since the original version:
1) I set CHECK_LIST_DSBL_ORG to “0″ by default, based on ioerror.us’s comments here
2) I fixed the require list at the top of the file to include “LWP::UserAgent” after receiving a problem report from Chris.
You can download the plugin here: mt-proxyplug.pl.gz (2.5KB, gzip)
Update 2005/01/15: Please consider using mt-proxyplug instead of this plugin below.
Along the same lines as yesterday’s SpamAssassin and MovableType entry, here’s another weapon against the “texas hold’em” and “football–betting” idiots.
Those morons have a tendency to conceal their identity. They are hiding behind public proxy servers and bombard your servers with their crap from there. So all we have to do is to check for a public proxy when somebody tries to submit a comment. If I detect a submission through a public proxy server it will not end up on the site, but will be ignored silently.
The proxy check is done in two stages:
If a comment submission is coming from a public proxy we drop a line in the server’s error_log. Here are just some of the entries I captured since I installed it:
[Thu Jan 13 08:35:21 2005] comment denied - using proxy: 216.49.49.118 80.25.156.151 - poker/bushmills1614@rocketmail.com/80.58.4.111 [Thu Jan 13 08:35:46 2005] comment denied - using proxy: 24.215.40.47 - football betting/bob@y6322o.com/63.110.140.28 [Thu Jan 13 08:39:00 2005] comment denied - using proxy: 58.40.89.127 - phentermine/gocha9985@see.it/203.199.92.158 [Thu Jan 13 08:48:59 2005] comment denied - using proxy: 115.120.174.78, 127.0.0.1 - online poker/absolut5129@freemail.com/80.255.49.222
And before somebody points me at Brad’s DBSL plugin: I’ve tested all the proxies listed above and they do not appear in the DSBL.
In order to use the plugin you will need to have the perl module HTTP::CheckProxy on your system. Drop the perl-code below into your MT plugins folder and you should be ready to go. Again, this has only been tested under Apache!
You can download the compressed version here: mt-commentproxyblock.pl.gz (1 Kb,gzip)
Comments(2)
#!/usr/bin/perl -w
package MT::Plugin::CommentProxyBlock;
use strict;
use lib ‘../lib’;
use vars qw ($VERSION);
$VERSION=’0.2′;
use constant ACCEPT_RESPONSE => 1;
use constant DENY_RESPONSE => 0;
use MT;
use MT::App::Comments;
use HTTP::CheckProxy;
eval{ require MT::Plugin };
unless ($@) {
my $plugin = {
name => qq{Comment Proxy Block for Movable Type v$VERSION},
description => qq{Will block attempts to post a comment via a proxy server},
};
MT->add_plugin(new MT::Plugin($plugin));
# tell MT that we want to be called to filter comments
MT->add_callback(’CommentFilter’, 1, $plugin, \&proxyCheck_filter);
}
# proxyCheck_filter
#
# checks environment for an entry which indicates we are handling a request that
# came from a proxy server (HTTP_X_FORWARDED_FOR). If environment does not give
# any indication, check REMOTE_ADDR and see if it proxies requests for us. If
# either is true we deny the comment posting attempt. Tested in Apache only!
sub proxyCheck_filter {
my($eh,$app,$comment)=@_;
my($isProxy,$proxy)=(0,”);
# uncomment to get a complete dump of the environment
# dumpEnv();
# check environment
$proxy=$ENV{HTTP_X_FORWARDED_FOR};
if(defined($proxy) && length($proxy)) {
print STDERR “[”.scalar(localtime()).”] proxy request forwarded for: $proxy\n”;
# if we have a X-Forwarded-For header, it was most likely
# added by the system that sent the request
$proxy=$ENV{REMOTE_ADDR};
$isProxy++;
}
# check remote address
unless($isProxy) {
$proxy=$ENV{REMOTE_ADDR};
print STDERR “[”.scalar(localtime()).”] probing for open proxy: $proxy\n”;
my $p=HTTP::CheckProxy->new($proxy,qq{http://www.google.com/});
$isProxy++ if($p->guilty());
}
if($isProxy) {
print STDERR “[”.scalar(localtime()).”] comment denied - ” .
”using proxy: $proxy - ” .
join(”/”,$comment->author,$comment->email,$comment->ip) .
”\n”;
return DENY_RESPONSE;
}
return ACCEPT_RESPONSE;
}
sub dumpEnv {
print STDERR “[”.scalar(localtime()).”] environment for $0:\n”;
foreach my $key (sort keys %ENV) {
print STDERR “[”.scalar(localtime()).”] $key = “,$ENV{$key},”\n”;
}
}
1;
Take a look at how the current administration is spending your tax dollars to educate our children: The Content of Federally Funded Abstinence-Only Education Programs (PDF). I can’t believe that they get away with it.
Update 2005/01/15: Please consider combining the plugin below with mt-proxyplug for best results.
A few days ago I saw the post on ioerror.us which details a solution to link WordPress’s comment checking system with Spam Assassin. I run MovableType and a WordPress solution does not work for me. The code needed to change a bit before it was usable on my system.
After enabling it last night and disabling mt-blacklist, I’m happy to report that it has caught every single comment spam attempt (a total of 32 attempts were registered). Spam indications appear in my server’s error_log like this:
[Tue Jan 11 08:33:34 2005] spam from diet pills/jane_doe7082@work.com/ 148.244.150.58: score 10.7 (limit 5.0)
A message like this indicates that the ‘CommentFilter’ implemented in mt-spamassassin.pl has received notification from the Spam Assassin daemon that the current comment is over the Spam Assassin threshold.
In order to use the mt-spamassassin.pl plugin you will need to have Spam Assassin’s spamd running on your own network or need access to spamd running on a remote system. Enter the name of the system that runs spamd in $sa_spamd_host (use ‘localhost’ if it’s running on the same host as MovableType) and also enter the port number where spamd can be reached in $sa_spamd_port. And because I did not find a way to retrieve a blog owners email address from within the MoveableType plugin, please also enter your email address in $mt_owner. For SpamAssassin’s user_prefs to work, you should also set your real (unix) userid in $mt_userid. Drop the modified file in your blog’s plugins folder and it should be ready to go.
Thanks to http://www.ioerror.us/ for the cool idea!
You can download the compressed version here: mt-spamassassin.pl.gz (1.5 Kb,gzip)
Update 01/14/2005: I’ve since added another plugin called mt-commentproxyblock, which has detected every single spam submission on 01/13/2005 before it was passed through mt-spamassassin. It seems that the majority of spammers do use public proxies and those are easy to detect.
Update 01/20/2005: I just posed a new version of the plugin with a few enhancements. If you have both mt-spamassassin and mt-proxyplug on your system, a comment will be shortcut if mt-proxyplug has already determined that it comes from an open proxy. Specifically, mt-spamassassin will look at the visible-flag of the comment and will not work on comments which are not visible. This will cut down on processing time for spam comments.
Second, Justin was nice enough to correct the fake Message-header I’ve been sending to spamd to make it more RFC-2822 compliant. Thanks!
Third, you can now specify a $mt_moderate threshold value. This means that if a comment submission is below the Spam threshold (defined in Spam Assassin), but above the $mt_moderate value, it will be moderated instead of being allowed all the way through to the blog.
#!/usr/bin/perl -w
package MT::Plugin::SpamAssassin;
use strict;
use lib '../lib';
use vars qw ($VERSION);
$VERSION='0.4';
# (CHANGE ME) what host is running spamd?
my $sa_spamd_host = q{localhost};
# (CHANGE ME) what port is spamd listening on?
my $sa_spamd_port = 783;
# (CHANGE ME) who is the owner of the blog?
my $mt_owner = q{me@localhost.com};
# (CHANGE ME) what is the userid for SpamAssassin?
my $mt_userid = q{me};
# (CHANGE ME) what is the moderate threshold?
my $mt_moderate = 1.5;
use constant ACCEPT_RESPONSE => 1;
use constant DENY_RESPONSE => 0;
use MT;
use MT::App::Comments;
use IO::Socket;
use Time::Local qw(timegm);
use POSIX;
eval{ require MT::Plugin };
unless ($@) {
my $plugin = {
name => qq{Spamassassin for Movable Type v$VERSION},
description => qq{Spamassassin for Movable Type},
};
MT->add_plugin(new MT::Plugin($plugin));
# tell MT that we want to be called to filter comments
MT->add_callback('CommentFilter', 10, $plugin, \&sa_filter);
}
# sa_filter
#
# 'CommentFilter' that is called for each attempt to post a comment
# on your blog. We'll pass the incoming comment to spamd running on
# $sa_spamd_host:$sa_spamd_port. If spamd responds with an indication
# that the comment was spam, then we'll repond with DENY_RESPONSE.
# If spamd says it's no spam or we can't get a good connection to
# spamd, we'll respond with ACCEPT_RESPONSE
sub sa_filter {
my($eh,$app,$comment)=@_;
unless($comment->visible()) {
return ACCEPT_RESPONSE;
}
#print STDERR "[".scalar(localtime())."] mt-spamassassin: " .
# join("/",$comment->author,$comment->email,$comment->url,$comment->ip) . "\n";
my $now=rfc822_date();
my $hostname=gethostbyaddr(inet_aton($comment->ip), AF_INET);
my $message="From " . $comment->email . " " . $now . "\n" .
"Received: from client ([" . $comment->ip . "] ".
($hostname?$hostname:$comment->ip) . ")" .
" by " . $ENV{HTTP_HOST} . " via MovableType; " . $now . "\n" .
"Message-id: <". sprintf("%x\$%x",time,rand(65535)) .
"\@" . ($hostname?$hostname:sprintf("[%s]",$comment->ip)) . ">\n" .
"From: " . $comment->author .
" <" . $comment->email . ">\nDate: " . $now . "\n" .
"Subject: MovableType comment\n" .
"To: $mt_owner\n\n" .
$comment->url . "\n".
$comment->text;
# make sure all lines end in "\r\n";
$message =~ s/\r\n/\n/gs;
$message =~ s/\r/\n/gs;
$message =~ s/\n/\r\n/gs;
# now send it off to Spamassassin
my $socket=IO::Socket::INET->new(PeerAddr => $sa_spamd_host,
PeerPort => $sa_spamd_port,
Proto => "tcp",
Type => SOCK_STREAM);
# no socket - no spam check
return ACCEPT_RESPONSE unless($socket);
# create the CHECK message for spamd
$message = "CHECK SPAMC/1.2\r\n" .
"User: $mt_userid\r\n" .
"Content-Length: ".length($message).
"\r\n\r\n".
$message;
# print STDERR "[".scalar(localtime())."] sending to spamd:\n$message\n";
# send it to spamd
my $toSend=$message;
while(length($toSend)) {
my $written = $socket->send($toSend);
unless(defined($written)) {
# oh no, something went wrong 
return ACCEPT_RESPONSE;
}
$toSend=substr($toSend,$written);
}
# close writing end of socket
$socket->shutdown(1);
# suck in response from SpamAssassin
my $response;
while(1) {
my $buffer;
unless(defined($socket->recv($buffer, 1024))) {
return ACCEPT_RESPONSE;
}
last unless(length($buffer));
$response .= $buffer;
}
# trim whitespace off the beginning of the response
$response =~ s/^\s*//;
# check if it is really a SpamAssassin response
return ACCEPT_RESPONSE unless ($response =~ /^spamd\/[\d\.]+/i);
# now find "Spam: True|False ; score / limit" header
return ACCEPT_RESPONSE
unless ($response =~ /spam:\s*(\S+)\s*;\s*([\d\.]+)\s*\/\s*([\d\.]+)/is);
my($flag,$score,$limit)=($1,$2,$3);
#if($flag =~ /false/i) {
#print STDERR "[".scalar(localtime())."] no spam:\n$message\n";
#}
print STDERR "[".scalar(localtime())."] spam $flag from " .
join("/",$comment->author,$comment->email,$comment->ip) .
": score $score (limit $limit)\n";
if($flag =~ /false/i) {
if($score > $mt_moderate) {
print STDERR "[".scalar(localtime())."] moderating comment\n";
$comment->visible(0);
}
return ACCEPT_RESPONSE;
}
# log a line to the error_log
return DENY_RESPONSE;
}
# rfc822_date
#
# generate a GMT date according to rfc822
sub rfc822_date {
# offset in hours (from Mail::Sendmail)
my $offset = sprintf "%.1f", (timegm(localtime) - time) / 3600;
my $minutes = sprintf "%02d", abs( $offset - int($offset) ) * 60;
my $TZ = sprintf("%+03d", int($offset)) . $minutes;
return POSIX::strftime("%a, %d %b %Y %T $TZ",localtime(time()));
}
1;
I know, I know - Christmas is already weeks past, we’ve taken down the tree already and I still have not posted any photos … sorry - it was busy around here with all the guests, parties and with my trip to CES in Las Vegas. But now I’ve found a few minutes to update you all on last year’s Christmas events.
We started off the day by unpacking the package that grandparents Sieglinde and Peter sent from Germany. It was a combination of Christmas/Birthday presents and CARE packet. CARE, because my parents sent a number of impossible-to-get-in-the-US delicacies from Bavaria. Foremost a huge bag of mum’s yummy, yummy home-made Christmas cookies (thank you very much again - those cookies were certainly one of the highlights of this Christmas). Here we have the little one digging into the package and finding the most important item immediately:
We decide to open the presents from-us-for-us in the morning to give Pia a chance to rip open some of the packages. She loves the mess that this generates and especially likes hiding in one of the boxes:
And because we have beautiful weather on 12/25/2004 we pack Pia in her snow-suite and take our three kids (Pia, Max and Frisco) out for a quick walk behind the house. It’s the first time for her to really walk in the snow and she can’t stop giggeling. When she falls flat on her face into the snow, she discovers that it is almost like ice-cream, without a specific taste that is.
After the walk we finish preparations for the Christmas lunch:
Candice and Pia pose in front of the tree and we all wait for the guests to arrive:
Candice’s parents Betty and Jim, Candice’s sister Pam, Pam’s fiance Edwin and Edwin’s mom Norma all joins us that Christmas day. We spend the rest of the day with a lot of eating, even more drinking and unpacking of presents.
There’s a reason why the following picture was turned black & white: because of the “drinking” mentioned above, some of the faces in this photo have a nice red glow in the color version. I thought it was more flattering to hide the facial discoloration in this family portrait …
And finally here we have Edwin, Pam and mom Norma:
Thanks everybody for making this a very happy Christmas day.
