For some reason, some how, the virtual private server that this site is hosted on was compromised by Baltic hackers last weekend. I am not sure what it is they wanted, maybe Trent Reznor or Stephanie Mayer have some irate technological fans out in Eastern Europe.
WARNING: Semi-technical talk ahead.
Either somebody took offense at something we have written or they were just really, really bored, because at 16:00 on Friday of last week our server started sending out massive amounts of data over UDP. I received an automated notice from Linode a few hours later telling me that my server had averaged 6.16MB/s over the last two hours. That one spike there managed to use up 5% of our monthly bandwidth. The network usage continued at random intervals for only brief periods of time, making it quite difficult to catch with tcpdump, and neither ntop or iftop noticed anything amiss.
Keeping an eye on Linode’s conveniently provided network graphs I was finally able to capture the tail end of another massive output with tcpdump only to reveal absolutely nothing. None of the (very helpful) Linode staff were able to tell me just what was going on, but after a few hours came to the conclusion that it was likely somebody had compromised my server and was probably using it to perform DDoS attacks. Solution: delete and rebuild.
This could’ve been painful if I was still with Media Temple, but Linode offer pro-rata billing (meaning that I only have to pay for what I use). I immediately opened up a new server and got to work transferring my backups. Then it was time to secure it, and better than I had done last time. I am now using iptables to block all ports other than www, https and ssh. I have also set up SSH public/private keys, disabled root login, moved ssh ports, removed vsftpd and installed fail2ban.
I could now close up my old account. Total cost to me: $0.63 in server charges and half my weekend. Hopefully I won’t have to deal with this again, but it taught me to make sure I always have up-to-date backups and to keep on top of security. And not to piss off Eastern European hackers.