Note: This article refers to an older version of Acunetix. Click here to download the latest version.
When you visit a website your browser sends an HTTP header called “User-Agent” to the web server. This header indicates which web browser you are using, its version number and details about your operating system and version.
Various browsers send different User-Agent strings. For example, Internet Explorer 9 sends Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0). If you are using an iPhone 4, for example, you will have a User-Agent similar to this one: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7.
In order to improve the user experience, more and more websites display one version for users who access the website from their mobile devices and another version for users who access the website from their desktop computers. When accessed, these websites automatically know if you are using a mobile as they parse the User-Agent string. Also, some websites show some content when visited by Google, while showing other content to regular users.
For example, if you visit Facebook from a regular desktop computer you will see this page:
However, if you visit the same page from an iPhone, you will be redirected to a mobile version of the site that looks like this:
One of the new features in Acunetix Web Vulnerability Scanner 8 is crawling websites and automatically using various User-Agents during the same crawl. This allows you to discover far more content and vulnerabilities. To demonstrate this, we’ve built a simple website that will show the user different content based on the User-Agent string being used.
When we crawled this website with Acunetix WVS 7, we could see the below limited website structure. This is because Acunetix WVS 7 was using a fixed User-Agent throughout the entire crawl process and therefore it did not crawl the “different” versions of the website.
When we crawled the same website with Acunetix WVS 8, we could see a complete website structure. The crawler from WVS 8 will crawl the website with various User-Agent strings, (for example the default one, the iPhone User-Agent and the Googlebot user-agent) and will follow any new links with the original User-Agent.
The website is not just crawled using different User-Agent strings, but it is also tested with the User-Agent that it was discovered with. Here is one Cross-Site Scripting vulnerability (XSS) that was found with Acunetix WVS 8.
In conclusion, crawling a website using different User-Agent strings helps Acunetix WVS 8 to find more content (targeted to mobile users and/or Google) and discover more vulnerabilities.
To stay up to date with the latest web security news like the Acunetix Facebook Page, follow us on Twitter and read the Acunetix Blog.
Get the latest content on web security
in your inbox each week.
THE AUTHOR
