How to Recover from a Hacked Website Event

Any fellow website owner or webmaster you may ask who is beyond the novice stage will agree that one of their top priorities will always be keeping their websites secure. However, exploits and tools available to hackers are so vast, and software technologies evolve so rapidly, that it is very possible, maybe likely, that you will experience a hacked website.

This is even more probable if your website runs an open-source CMS such as WordPress or Joomla! with a lot of plugins. In particular, new WordPress security issues appear quite often and hacked WordPress sites are not a rare sight at all.

When addressing such an event, it can be helpful to have a short checklist of tasks to perform as part of your recovery process. Doing the right things in the right order will be key to maximize your chances of successful and complete recovery, as well as mitigation of future events.

Your ToDo List

Your ToDo list will contain two types of tasks: Preparation tasks and Action tasks. Preparation tasks involve no changes to your web site or any related or underlying components at all.

It is essential to clearly understand this point because your preferred first action must be to make sure that the attacker has no way to continue accessing the system; any other action that changes the web site may alert the attacker that they have been discovered before access has been blocked, and you do not want to trigger the attacker into either perpetrating more damage or covering their tracks.

Remember: once the event has happened, it must be treated not only as a reason to fix but equally as a motivation to harden and secure.

• Prepare: Reaction plan
• Prepare: Battle sheet
• Action: Take your system offline
• Prepare: Clone your system to a testbed or staging server
• Prepare: Scan your website for vulnerabilities; identify and confirm suspected intrusion point
• Action: Fix the vulnerability
• Action: Bring the fixed version of the site back online; whenever possible, you should redeploy the sanitized version of your website to a clean OS/web server setup
• Prepare: Monitor your new and improved website
• Prepare: Make a reaction plan for future events

Prepare: Reaction Plan

Keep calm and focused, and acknowledge that the event may have happened some time ago; it will not matter if you react after 30 seconds or after 15 minutes.

Once your web site has been hacked, taking a few minutes to think things through to get the most out of the incident is absolutely necessary. Take the time to write down all the steps you need to take to:

• Preserve the current situation of the web site and its underlying data to give you the best chance to eventually understand how the breach was enacted
• Avoid making rapid changes that will alert the attacker to cover their tracks or cause more damage
• Collect information and tools necessary to administer the web site and related components.

Prepare: Battle Sheet

If you are going into battle, you must make sure your armory is populated with a complete arsenal of weapons. Basically, this means having at hand:

• Access details and relative credentials for your web site
• Filenames and paths for any configuration files, which may contain information that governs access to the web site itself, or to other systems, which your web site connects to or integrates with, such as:
  • Database access details and credentials
  • Third-party systems integration details and relative credentials, API keys, or security tokens
• Access details and relative credentials for the OS or platform underlying your web site
• If you are managing the underlying hardware, access details and credentials for the “lights-out” access tools (DRAC, ILO, KVM, local or remote console, etc.)
• Telephone numbers, email addresses, and support portal information for any vendor providing you with the web server and any of the underlying layers, including any related identification tokens or PINs/telephone numbers, email addresses for the developer resources you will engage for any eventual code changes or other programmatic fixes to your website.

Action: Take Your System Offline

Your first action should be to take your system offline. The benefits of this are:

• Limits any additional damage the attacker may inflict on your website
• Prevents the attacker from covering their tracks or hiding any evidence which may lead you back to them, or to their methods, or both
• Prevents your system from inflicting additional damage on other people, as well as accessing or damaging data or scripts on other websites (spam, Cross-site Scripting, and so on)

Before you can take your system offline, make sure that you have carefully compiled your Battle Sheet with all the necessary information for you to continue accessing the system, even though it will be offline.

Specific steps that you need to do to take your system offline depend on whether you are using a hosting company or whether your web host is located at your company premises.

Prepare: Clone Your System to a Testbed or Staging Server

Some of your investigation tasks will involve running tests on your website to understand where and how it can be penetrated (hence the coined phrases “pen test” and “pen testing“). Some of these tests can be aggressive and either cause damage to the web site or to its underlying data or in this case possibly to any artifacts left on the system by the attacker, which could lead you to their identity or methodology or both.

To eliminate the effects of these risks, you will clone your system to a completely separate testbed or staging server. This testbed should not be located at a hosting provider because it must be isolated from any public IP address space to make sure your cloned system cannot be reached from the outside, and that your cloned system cannot itself access or affect third parties. Any systems used to test the cloned system would connect to the same private IP address space as the clone for the purpose of running the tests.

Prepare: Scan Your Website for Vulnerabilities

Once your testbed is populated with a clone of your website and related components, you can safely test your system.

You would typically use a two-pronged approach, running simultaneously:

• Scan your entire website for all possible points of entry
• Look for this event’s specific point of entry

Scan Your Entire Website Using Automation

The first channel of investigation would be to use a system audit approach – using a reputable web vulnerability scanner, launch a scan on your cloned testbed, which should perform the following:

• Identify the misconfiguration of the web server and related software
• Identify old and known vulnerable software versions (web server software and operating system patches are typical suspects)
• Identify insecure security access methods (for example, weak passwords susceptible to brute force or dictionary attacks or weak cipher mechanisms)
• Identify inappropriate access to filesystem locations (such as directory visibility and permission issues)
• …and run a very large battery of specific tests looking for specific known vulnerabilities

This scanner will provide you with a list of vulnerabilities that your website is susceptible to. This type of scan can take a considerable amount of time, particularly for large and complex websites. You can use this “waiting” time to…

Look for a Point of Entry Manually

It is critical to find the actual point of entry that triggered this event. Even if you were to do a complete restore from a backup that would eliminate any damage caused by the attacker, it is only reasonable to assume that the attacker can very rapidly gain access again using the exact same vehicle.

Where to look? This is not an exhaustive list, and indeed as you grow in experience and get a deeper familiarity with the specific systems you are managing, you will be able to build a more comprehensive and specific list for your needs. The list below, however, presents a usable starting point. Do keep in mind that this particular list’s investigative actions are performed on the real web server, not on the testbed, so for the time being it is important that your maneuvers are limited to investigation and information collection – do not make any changes for the time being.

• Get a list of running processes and look for suspicious running applications or services such as backdoors or trojans
  • If you have a clean previous version to compare to this would be very helpful
  • Identifying the file path for a suspicious running malware program may help you identify the point of entry (you may want to use antivirus software to help)
  • Identifying the user under which a suspicious program is running or the user account or group that owns the program may help you identify the point of entry
• If your system has been hacked, there is a very large possibility that the attacker may have introduced new files with malicious code or changed already existing files on your system
  • If you know the date of your last change to your system, you may simply get away with checking all the files on your system to find those that have been modified or created after your last changes
  • If your web server’s file system is static, it may be useful to compare it to an older documented file system dump; if you do not have one, then your final review may want to consider this step for any future policies revolving around web site security
  • If not already installed, you may consider deploying software to your web server that regularly checks file system contents against a known clean file system list (typically called a baseline); this concept is called HIDS, or Host-based Intrusion Detection System; you can start by reading up about AIDE, syschangemon, Mugsy, Samhain, as a starting selection of open source tools to implement on your web server and use Google search engine for more information
  • Check out the user or group that owns the created or modified files since this may be helpful to identify the point of entry
  • Carefully check out the contents of created or modified files; the contents added or changed may help you identify the perpetrator and/or their methods, or possibly help you understand if your website is being used as a platform to launch an attack on other third party websites
• Look at the log files generated by the operating system, the web server software, and any other software also used by your web site (database servers for example)

Hopefully, your painstaking investigation of log files, running processes, and anomalous filesystem content will have allowed you to identify the security flaw and the entry point.

Compare Manual Results with Automated Results

If you have identified one or more suspect points of entry, it can be interesting and instructional to compare your findings with the list of vulnerabilities identified by the automated web vulnerability scan.

Action: Fix the Vulnerabilities

Using the information gained from your manual investigation of files, processes, and logs, together with the (hopefully short) list of vulnerabilities compiled by the web vulnerability scanner, you can now proceed to implement the code improvements or server updates and hardening to prevent such an event from repeating itself and plug any other weaknesses, generally making your web site safer. Summarised briefly:

• Stop any foreign processes
• Remove any extraneous files or executables
• Replace changed files and data sets with fresh or sanitized versions

This might seem obvious, but it is critically important to check any fixes implemented by repeating the automated scan, thereby confirming that the fixes are in fact effective.

Action: Bring Your Fixed Web Site Online

Now that the clean up is completed, suspect files and processes have been stopped and eliminated, the points of entry secured, and other security measures are in place, you can bring your web site back online.

If you have any doubt about having fully contained and reversed the work of the attacker, you should deploy a new web server host and redeploy your sanitized website and ancillary components to this new host. If you run an open-source CMS such as a WordPress website, it may even be a good idea to install the newest version from scratch with a limited number of plugins (also using their newest versions).

Prepare: Monitor Your Website

Your website is back up and running. Do not breathe too easy, however, because you need to make sure that your fixes are effective. In particular, you should devise some way (by monitoring log messages for example) to have an early-warning system of an attacker trying to get into your system via the same access vector as the event you have just fixed.

If the attacker does manage to get back in, then either you have not identified the entry point correctly, or you have not identified all of them, or the fix is inadequate; in any case, you will need to repeat the process to again recover from this latest hack event.

Prepare: Reaction Plan for Future Events

After you have completely recovered from this event and completed all the tasks related to this event, you should revisit this section, review the Reaction Plan you made for this event, and use it to build a recovery procedure for any future event.

This will allow you and other members of your support team to more effectively and efficiently handle your next event.