Barracuda Networks Breached

Introduction

On April 11th 2011, at nine in the evening, Barracuda Networks posted a grim entry on their blog. Their network had been hacked. Thousands of their confidential customer and employee records were stolen. In an ironic twist of fate, the company that advocates security through it’s own Web Application Firewall were victims to the most common and oldest type of attack against web servers – the infamous Blind SQL Injection.</>

Web Application Firewalls like the one that Barracuda Networks manufactures are designed to stop this type of attack from occurring in the first place, however they clearly are not the silver bullet against hackers. On their blog, Barracuda Networks admitted that they made several mistakes, the biggest of which was to unintentionally turn off their own firewall for a few hours. This was a golden window of opportunity which hackers pounced on and immediately exploited.

In a vain attempt to regain their customer’s trust, Barracuda Networks explained how the vulnerability happened. They mentioned that no sensitive information was stolen and all the stolen passwords were salted. The skeptic in me decided to challenge their statements so I did my own independent research, got hold of the stolen data and began an analysis.

This article details my findings, where I discover that within the 20,000 records that were stolen, many were internal and possibly privileged user names and passwords. I also find that some of the most sensitive passwords were very easily cracked and were not salted as they claim. I also challenge the assumption that Web Application Firewalls can never be one’s only line of defense. In fact, after this breach I am tempted to discourage the use of such devices since they easily lead to a sense of false security.

Main Concepts Behind the Attack

In this section I briefly explain the different methods used during this attack. I appreciate that some of you will already be familiar with these concepts, however it is good to go over them because it will give you a better understanding of the technical analysis later.

Blind SQL Injection

An SQL Injection attack is executed in three phases. In the first phase, the attacker launches a series of probes, or scans against his target. These scans are testing for any known SQL Injection weakness. They typically work by sending intentionally malformed user data to the server and analysing error responses from the web application. Certain error responses pinpoint vulnerabilities, whilst others reveal important information which is used to further refine the scans.

In a Blind SQL Injection attack the web application does not reveal any information about the errors, therefore traditional probing methods are ineffective. This does not mean that there are no vulnerabilities, but it does make the existing ones much harder to find.

Web Application Firewalls

The concept behind a Web Application Firewall (WAF) is very similar to how a traditional firewall works. A set of rules are configured on the firewall that selectively allow or deny network traffic. Rules on a WAF are exclusively designed to filter HTTP traffic. They are also capable of detecting common attacks, such as SQL Injection attack probes and XSS attempts. These firewalls exist as software installed on a host, or as a dedicated hardware device. Web Application Firewalls are a good first line of defence, however this does not mean that web application developers can employ lax coding and testing standards, as application bugs can still be exploited through the firewall. Barracuda Networks are a major player in the field of Web Application Firewalls, however they still fell victim of the classical SQL injection attack.

Hashing Passwords and Salting Hashes

Hashing passwords is a popular alternative to encrypting and storing passwords in a database. A hash is a one-way algorithm. This means that a password can be turned into a hash, but this a hash cannot be turned into a password. This is very convenient for web logins. The web server never stores the password, but instead stores the hash of the password. When a user attempts to log in, the password he enters is turned into a hash and compared with the hash in the database. If the two hashes match he is authenticated.

Cracking Hashed Passwords

Cracking hashes is only possible with brute force. A hacker tries every possible password combination, turning each password into a hash and comparing it to the hashes he has stolen. If one of them matches, then he has found a password. This attack, although cumbersome is greatly facilitated by the use of rainbow tables. These are large files of precomputed hashes for millions of known passwords and greatly reduce the time needed for a brute force attack.

To protect against rainbow table attacks against hashes, security administrators are encouraged to salt their passwords before hashing. This is a simple and effective technique. It consists of adding a special value to the password before hashing and renders brute force attacks practically useless.

Anatomy of the Attack

The news of this breach begs the question: How does one get past a Web Application Firewall (WAF) and perform an SQL Injection attack? The answer is simple. Wait for the WAF to be disabled and then perform the attack.

Below is a diagram showing how a WAF is positioned to block these attacks before they even reach the web server.

In the diagram above you can see that legitimate traffic is allowed through the WAF, however an SQL Injection payload is stopped before it can reach the Web Application Server, and consequently the SQL database.

If the WAF is removed, however the Web Application Server is left exposed, as can be seen in the diagram below.

According to various reports, attackers were able to exploit this scenario for many hours until they were successful. The SQL Injection vulnerability itself was found in a PHP script which is used to list customer reference case studies. Although this script was designed to access only the partner database, the SQL breach was large enough to give the hackers access to other databases on the same system.

According to a statement issued by Barracuda Networks the attack started with a few hours of probing from one IP address which was followed by a full-blown attack from several locations.

The vulnerable URL was the following:

http://www.barracudanetworks.com/ns/customers/customer_verticals.php?v=11

As you can see from the URL above, a single parameter called “v” is passed to this module. This parameter, which is of numeric type, identifies a vertical market ID. Vertical market  ID 11, corresponds to “Entertainment and Leisure”. Changing this ID to 10 serves up the vertical market “Energy and Utilities”.

By manipulating this parameter, hackers were able to inject their own SQL commands into the system and read out the entire set of databases on the server.

The hackers also revealed some other information, which is shown below:

It appears that the web server was running on a Microsoft Windows platform running the Microsoft IIS 6.0 web server. The vulnerable web application is listed as ASP.NET which looks odd to me, since the vulnerable URL is a PHP script and not an ASP script as one would expect on a Microsoft ASP.NET box.

The total extent of the breach may never be known to us, however some interesting information can be garnered through an analysis of the information leaked by the hackers themselves. The next section sifts through this information to determine what was stolen.

Damage Suffered

The hackers who claim to be responsible for the breach are a Malaysian group that go by the name HMSec. They posted a Full Disclosure message a few hours after the breach. In this message they listed a total of 22 different databases. Some databases are well known such as “phpmyadmin”, a popular administration tool for PHP and MySQL, who had also been a victim of an SQL Injection hack some time ago. I also noticed a database called “php_live_chat” which could mean that they are running a product called phpLiveChat, a commercial software module that allows customer to interact directly with sales and support staff at Barracuda.

There are also some more interesting databases. For example, the database “information_schema” is probably loaded with information that the hackers could have used to penetrate the database even further. There are two database called “bware” and “black_ips”. I hope my IP addresses are not in that one!

I saw a database or two which look like they belong on a development machine and not a live web server. Some examples of these are “igivetest” and “igivetestsucks”. A google search for “igivetest” results in an online tool for creating multiple choice questions. I wonder why this software was being installed and experimented with on a live server. Other questionable databases are “dev_new_barracuda” and “new_barracuda_archive”, both databases look out of place and make for a very dirty mySQL implementation on their end.

The juicy parts are what come next, list upon list of user names and passwords.

Here are a few, which I have censored out of respect for Barracuda staff, partners and customers.

This table is called CMS_LOGINS and contains 251 login accounts for the Barracuda Content Management System. The hashed passwords use the MD5 algorithm.

The following passwords were extracted from the main mySQL database itself and contains users who have system-level privileges some which are tied down to particular servers and some which grant access to all servers on the network. This database contains 23 users.

The users below were extracted from another database and contains user names, passwords and email addresses of Barracuda employees, possibly those who had access to the web help-desk system.

As you can see from the above tables, none of the passwords were stored in clear-text. Barracuda attempted to downplay the importance of these hashes by saying that they were salted, however upon closer inspection I found this claim to be unfounded.

Take a look at the main mySQL table. It contains duplicate hashes which not only shows that the administrator was using the same passwords for different accounts, but also shows that salting did not take place, or did not take place properly. Salting should append a different value to every password before it is hashed therefore making each of the hashes unique which is clearly not the case here.

After looking at this, I decided to try a rainbow table attack on some of the hashes. A rainbow table attack is very simple; huge lists of precomputed hashes are obtained (Google is your friend) and a simple text search is done within them. The crudest way of doing this is to simply paste the hash into Google and if you are lucky, you will come up with a match. Salting makes this trivial attack useless, so if the passwords were salted I would not have been successful.

For this test I took the hashes from the help-desk administrators table, below are some of the results:

The password above, which took less than half a second to crack is “zombie”, hardly a safe password. The result below makes me want to cry.

You are not seeing it incorrectly. It really does say that this password is “password”, the least secure password thatcan ever be used. How can Barracuda Networks risk such embarrassment?

Some other passwords were not as easy to crack, however I succeeded anyway after enlisting my Romanian friends who helped me brute-force some of the stronger passwords.

Below are some of the results after about 50 hours of brute-force cracking. These hashes come from different databases that were breached during the attack.

Needless to say, these passwords were not strong enough to withstand a brute-force attack and contrary to Barracuda’s false claims, they were obviously not salted either.

Lessons Learned

In a blog post to the Barracuda Labs security blog Micheal (Mike) Perone listed a few lessons that his team learned after the breach. In summary, they are:

You can’t leave a Web site exposed nowadays for even a day (or less).

Code vulnerabilities can happen in places far away from the data you’re trying to protect.

You can’t be complacent about coding practices, operations or even the lack of private data on your site – even when you have WAF technology deployed.

The first two lessons are trivial and probably show the naivety of Barracuda staff. A website cannot be exposed even for a few seconds, let alone a day! Furthermore, in this globalised and networked world, “places far away” can never be more than a few milliseconds apart.

I do applaud Mike on his third point. WAF technology is not enough. It should never be considered to be the only line of defence. At best a WAF will mitigate the risk of attacks, and if configured properly can reduce the load on your web servers. At worst (as in this breach), a WAF can fool you into thinking that you are safe enough, and therefore deeper tests such as code reviews and scans performed by a web vulnerability scanner might be neglected. This leaves your web application exposed to threats, and should the WAF fail, vulnerabilities will stick out like a sore thumb.

In addition to the lessons learned, I would also like to add a few of my own. Hopefully someone working at Barracuda will be humble enough to listen to me and learn something.

Be honest. Credibility with your customers is important, and your customers are not fools. After a breach like this Barracuda should have never made false claims that could be refuted using a simple Google search.

Salt your passwords. Really. Do not just wish or pretend you did. Salting is easy, recommended by everyone and extremely effective in mitigating attacks. It will also save you a whole lot of embarrassment when your administrator uses passwords like “zombie” and “password”.

Have a password policy in place. Do not allow easily guessable passwords to be the weak link in your chain of security. Password policies are easy to enforce through software and although are a nuisance to your users they remain a necessary evil.

Use a vulnerability scanner. Do not hide behind a Web Application Firewall. Scan your networks instead and identify bugs in your web applications. When you expose yourself you inevitably tend to protect yourself better.