Tales from a base64 wordpress hack, part 2: recovery

posted in: Tech | 7

Time for some investigative work. This post may get unapologetically technical.

So I began by looking at my access logs from the day that the compromise occurred, I started with just a full browse of the log to look for anything anomalous. Discovered a lot of hotlinking to images on my site! Squashed that problem (go ahead and try it! You might have to clear your cache first to see the effects….)

I suspected that looking for POST requests would be a good place to start since there were likely to be fewer of them.

Which revealed, among other things:

In the previous post, I mentioned the backdoors — well here it is! So it looks like the attack vector was via a backdoor uploaded to the current theme. I went and scanned that directory:

So the backdoor was installed 2 days prior to the attack. The attack itself occurred yesterday, so it’s good to know that it was a quick turnaround from detection to removal, and also good to know they only had 2 days to do damage. At this point, I now knew which date to look for, so I looked in the access logs from the 27th.

Tried the following:

Then I remembered that since WordPress requests are done using pretty URLs, the extension “php” shouldn’t be mentioned anywhere in the requests, so I tried: cat access.log.2012-02-27 | grep php

Bingo.

Scanning through the results, I see these three lines:

The interesting thing here is that the attacker tried three different places, though it appears it was all in one second of one another, so it’s also possible there were three separate files. The installer12.php file was removed by the time I looked for it. Just to be I searched: find ~/ -name installer*.php but found nothing.

Immediately after this line, I saw two requests to wp-cron.php, which I had been previously dismissing as maintenance, since the IP address was dreamhost. Up until now, I had been assuming this was regular maintenance. But the fact that it specifically pointed to the theme that was the point of entry for this hack, I decided it was worth investigating.

Quick search on google for installer12.php — shows up in some random foreign websites in very weird places, like /images/. Something is fishy here. This means two things:

  • The attacker was able to spoof my host or had shell access
  • The attacker was using wp-cron.php to do the dirty work

I also found an article on wordpress.org about others who had similar problems, and came to the same conclusion.

At this point, I need to look at wp-cron.php and find out what is referencing cron jobs: where doing_wp_cron equals 1330393678 and 1330402052.

Browsing into wp-cron.php source, after verifying that mine matched the canonical source,   it appears that the numbers are merely timestamps and may not be uniquely useful in themselves; HOWEVER, it does mean that I need to look at the cron table for WP, which is loaded by _get_cron_array(). After some digging, this appears to be the query that is ultimately pulling data from the DB:

So, time to investigate the options table for this blog.  There were a few things I expected to see (scheduled, scheduled_delete, etc.) and then two rogue entries:

Now I don’t know for a fact that these entries are junk, but they seem odd. The rest of the options table looked legit. Not enough to go by here.

Back to the log.

I thought this time I would filter based on finding “minicard”, the theme that was first compromised. (Backdoor was created at 4:37 on 2/27, so anything nefarious before that should be useful).

After the installer was run, this shows up in the logs:

The string: …jquery.fancybox/$%28this%29.attr%28 evaluates to: jquery.fancybox/$(this).attr( — which is jQuery. This looks syntactically similar to an open-ended SQL injection attack.

Under the assumption that the attacker might have uploaded the payload before injecting my host to launch it (and so before the exploit popped the IP address would be different), I searched instead for: 

The year and hour (date is implied, since this is the log for one day only).

I’m seeing some curious traffic sourcing from what I was passing off as a robot before:

Now what’s interesting here is that the last entry’s IP matches with who it claims to be (spinn3r, a blagosphere indexing robot):

…but the first three don’t match.

Thinking that perhaps Majestic12 was some hacking group, a google search reveals a blog post from a couple years ago where many people were being crawled and attacked by someone pretending to be the M12 bot. Ok, now we’re onto something. I’ve at least been able to identify the attacker — all incidences of that M12 bot showing up in this time period should be considered hostile. That article does say that “1.4.x” should be a legitimate version of the bot, it was also from a couple years ago. Inconclusive.

UPDATE: I’ve determined that the attacks came from kasserver.com, though I don’t know if kasserver.com was complicit in the attacks.

Though the day prior to this one had additional hits come in from the M12 bot from a different IP, this one resolving to: 211.73.212.62.in-addr.arpa name = evo-hl21-1.gameservers.net.

More to come later. I need to research some more, and I’ve contacted Dreamhost support to see if they can pull anything from their logs.

7 Responses

  1. Steve

    Hi Aaron,

    My name is Steve, I work for Majestic-12 Ltd, who co-ordinate the MJ12bot. I found your post via google, and was really sorry to hear about your blog being hacked. I notice you’ve found MJ12bot in your logs, and I just wanted to drop a line to re-assure you that we are a legit company with no interest in hacking webmasters such as yourself. We run a distributed crawl, but do put lots of effort into obeying robots.txt.

    A contact email is available on the url referenced by the user agent – http://www.majestic12.co.uk/bot.php – Aaron, if you have any concerns, please drop us a line on the email there ( or the one supplied above ) and we will look into it – we take our crawl very seriously and are all too happy to investigate any concerns anyone may have.

    As a brief explanation of what we do – and I don’t want this to turn into an advert ( so feel free to delete this paragraph from the public blog if you want ), we run a distributed crawl, co-ordinated via our project homepage at http://www.majestic12.co.uk. currently, the crawl is used to power a map of the internet – data from which is available at http://www.majesticseo.com. We do offer free reports to webmasters – all we ask is they register onsite, and a simple verification file needs uploading to the root web dir to prove ownership.

    Please get in touch with us if you need any re-assurance what-so-ever.

    Kind Regards,

    Steve

    • Aaron

      Steve — I need to update that post, and will probably do so tonight — I believe the attacker came in on an obsolete plugin that I have since removed.

      I appreciate you stopping by though! I will update the original post to correct that.

  2. Alissa

    Great writeup on this Aaron. My sites got hacked with the same code in the last week. I found the same installer12.php file. It came from the same IP address as my host. That file was initially accessed on 2/26/12 at 14:01 PST.

    About 24 hours later, the backdoor was executed through a file called aggisherman.php.

    In my case, it looks like the original backdoor came in through a Mint Pepper that I had installed. Likely the permissions were set incorrectly.

    • Aaron

      Alissa,

      Please see my most recent post (part 3) for some other details about preventing future infections and preventative measures you can take to recover more quickly.

  3. Nima

    Thanks much for the writeup! I’m currently struggling with similar attack where malware was basically appended to a bunch of random js files on different WP installations on different domains on my server. I see similar suspicious entries starting July 11 like this one for example:

    POST /wp-cron.php?doing_wp_cron=1342088761.3603510856628417968750

    I’m trying to figure out how the attacked was able to write into js files on my server, will let you know if I find anything

    • Aaron

      I suspected those at first too — but if you check the source address for those addresses, it is likely the local IP, I think. I don’t *think* those are the attack vector.

      In all the cases I saw, it was timthumb.php exploits. Typically through URL injection — timthumb.php would write malicious code to the server, which could then be executed directly — this would set up the initial attack, and then they can connect directly via their Alucar (or whatever malware app they’re using) interface.

      The one that attacked my server was capable of date alteration, database access, filesystem manipulation upload/download, etc. You are probably best off by copying your wp-content folder to a separate place, scanning through the uploads folder (use the “find” command to find any files that are php, or rather, are not jpg, gif or png). Blow away your plugins and redownload, do the same with the theme. Better safe than sorry. :/