This site has been getting a lot of traffic from NewsRiver and StumbleUpon lately, which has created a bit of a quandary. As many of you know, I run ReferrerCop.org, a database of known referrer spammers. For the purposes of ReferrerCop, I've always defined referrer spam as "repeated website requests using a fake referrer URL". Usually, this is done by spammers who want to drive traffic to their sites, but in the case of NewsRiver and StumbleUpon, we have two services sending legitimate traffic, but providing fake referrer URLs.
On the one hand, this behavior is technically against the rules, and part of me wants to classify it as referrer spam. On the other, since the traffic itself is legitimate, webmasters may find it useful to see the referrals in their web statistics, even though the referrer URLs are just generic pages advertising the services. Personally, I find it annoying that these two referrers consistently occupy the top two slots of the referrer listing for wonko.com, yet at the same time I do like knowing that I'm getting a lot of traffic from them.
I'd like to hear what other webmasters have to say about this. Given the circumstances, do you think NewsRiver and StumbleUpon should be excepted from the rule, or should they be considered abusers just like anyone else who sends fake referrer information?
I'm proud to announce my latest attempt to improve the quality of your pitiful lives: ReferrerCop.org.
This is a companion website for ReferrerCop where you can download referrer spam blacklists in a variety of formats and submit your own web server log files to help fight the evil referrer spam scourge. You can also search the database of known referrer spammers, if that's the sort of thing you enjoy.
ReferrerCop 1.0.4 is now available. This release includes a minor bug fix and a new -U command-line option to check for updates to the default blacklist (which occur pretty frequently these days).
I'm still waiting for the rest of the world to discover ReferrerCop. I only know of a few people who use it, but it's probably the most useful piece of software I've ever written. I'm surprised there aren't ten other programs out there that do the same thing; instead, everyone seems to be misusing mod_rewrite and slowing down their web servers. Silly people.
I've released version 1.0.3 of
ReferrerCop. It's actually been mostly finished for a while now, I just hadn't gotten around to fixing one last bug and pushing it out the door. This release adds in-place filtering of files (which makes the program about 10 times more useful in my opinion) and improves the parsing of AWStats data files. You can download it
here or install from the
FreeBSD ports collection.
Recently there’s been an annoying rise in the amount of referrer spam targeting sites on my server, so I decided to do something about it.
ReferrerCop will parse your Apache log files or AWStats data files and remove entries for referrers that match a big list of known referrer spammers. The filtered files are returned via standard output. By running your logs through ReferrerCop either before or after they’ve been processed by AWStats (or the log analysis software of your choice), you can eliminate annoying referrer spam from your web statistics.
If you know Ruby, you can also use ReferrerCop’s filtering capabilities in your own programs simply by including the ReferrerCop module.
If enough people start using this, I’d like to add features for automatically updating the blacklist and for performing a wider range of tasks (including filtering more log file formats).
