For some reason, some how, the vir­tual private server that this site is hos­ted on was com­prom­ised by Baltic hack­ers last week­end. I am not sure what it is they wanted, maybe Trent Reznor or Stephanie Mayer have some irate tech­no­lo­gical fans out in East­ern Europe.

WARNING: Semi-technical talk ahead.

Either some­body took offense at some­thing we have writ­ten or they were just really, really bored, because at 16:00 on Fri­day of last week our server star­ted send­ing out massive amounts of data over UDP. I received an auto­mated notice from Linode a few hours later telling me that my server had aver­aged 6.16MB/s over the last two hours. That one spike there man­aged to use up 5% of our monthly band­width. The net­work usage con­tin­ued at ran­dom inter­vals for only brief peri­ods of time, mak­ing it quite dif­fi­cult to catch with tcp­dump, and neither ntop or iftop noticed any­thing amiss.

Keep­ing an eye on Linode’s con­veni­ently provided net­work graphs I was finally able to cap­ture the tail end of another massive out­put with tcp­dump only to reveal abso­lutely noth­ing. None of the (very help­ful) Linode staff were able to tell me just what was going on, but after a few hours came to the con­clu­sion that it was likely some­body had com­prom­ised my server and was prob­ably using it to per­form DDoS attacks. Solu­tion: delete and rebuild.

This could’ve been pain­ful if I was still with Media Temple, but Linode offer pro-rata billing (mean­ing that I only have to pay for what I use). I imme­di­ately opened up a new server and got to work trans­fer­ring my backups. Then it was time to secure it, and bet­ter than I had done last time. I am now using ipt­ables to block all ports other than www, https and ssh. I have also set up SSH public/private keys, dis­abled root login, moved ssh ports, removed vsftpd and installed fail2ban.

I could now close up my old account. Total cost to me: $0.63 in server charges and half my week­end. Hope­fully I won’t have to deal with this again, but it taught me to make sure I always have up-to-date backups and to keep on top of secur­ity. And not to piss off East­ern European hackers.