Last week, Larry Suto published a report entitled “Accuracy and Time Costs of Web Application Security Scanner Report”. I’ve started to investigate in detail the results from this report. And I’ve found a list of inaccuracies. Here is a direct quote from his paper:
Methodology
In order to cover as many bases as possible it was decided to run each scanner in two ways:
1. Point and Shoot (PaS): This includes nothing more than run default scanning options and provide credentials if the scanner supported it and the site used any.
2. Trained: This includes any configurations, macros, scripts or other training determined to be required to get the best possible results. As needed help was requested from the vendors or from acquaintances with expertise in each scanner to make sure that each was given all possible opportunity to get its best possible results.
Therefore he’s defining two modes; Point and Shoot and Trained. In the Point and Shoot mode he’s supposed to use the default scanning options AND provide credentials if the scanner supported it.
Except that for our scanner, he’s not doing this. Let’s take our test PHP website testphp.acunetix.com.
Here is a quick excerpt from his results:
Acunetix is listed as not finding any of the 4 XSS vulnerabilities from userinfo.php (trained or untrained).
That came as a big surprise to me. I’ve quickly made a test and surely, the vulnerabilities were found by Acunetix WVS.
This file “userinfo.php” is only available after you provide valid credentials, it’s not possible to access this file unauthenticated.
They were not found because Larry didn’t authenticated our scanner (didn’t provided any credentials). No wonder that Acunetix didn’t found the vulnerabilities. The same situation with the SQL vulnerability from cart.php (the shopping cart is only available when you are authenticated). He didn’t authenticated our scanner neither in the Point and Shoot mode or in the Trained mode. That’s not fair for us.
I then moved to the Cenzic test website (http://crackme.cenzic.com). Here Acunetix is listed as not finding a number of XSS vulnerabilities in various files such as /Kelev/php/transfer.php (parameters Amount, ToAccountNo), file /kelev/php/accttransaction.php (parameters FromDate, ToDate) and so on.
I’ve started a scan for crackme.cenzic.com and guess what? All those vulnerabilities were found by Acunetix WVS. I think it’s the same situation as before: the scanner was not authenticated and therefore, it couldn’t access those pages.
Below, I’ve attached a screen shot with those vulnerabilities found by Acunetix WVS:
Therefore, Acunetix WVS was clearly disadvantaged in this comparison report. It’s not possible to find vulnerabilities in authenticated pages without providing the right credentials.
In the end, I would like to point out a very suspicious log event from our test website. While analyzing the logs from testphp.acunetix.com I’ve found the following entry:
72.25.78.35 – – [20/Jan/2010:08:44:58 +0100] “GET /Flash/add.swf HTTP/1.1” 200 17418 “file:///C:/NTOBJECTIVES/SOURCE/ntospider_5_0/ntospider/NTOGUI/NtoGui/Debug/Reports/acunetix/
2010_01_19_23_43/DF4D21797A665BCA9B48B5B5F5C37C2” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618)”
This log entry was generated by NTOSpider while scanning our test website. What’s suspicious about this log entry is the Referer field:
file:///C:/NTOBJECTIVES/SOURCE/ntospider_5_0/ntospider/NTOGUI/NtoGui/Debug/Reports/acunetix/
2010_01_19_23_43/DF4D21797A665BCA9B48B5B5F5C37C2
Notice the directory: C:/NTOBJECTIVES/SOURCE/ntospider_5_0/? SOURCE? Debug?
Only NTObjectives employees should have access to the NTOSpider source code. I don’t have enough evidence to directly accuse NTObjectives, however, that log entry looks suspicious to me.
Get the latest content on web security
in your inbox each week.
THE AUTHOR
