WAVSEP2014
WAVSEP 2014 is an independent, up-to-date comparison of 63 black box web application vulnerability scanners and SAAS services, performed by the WAVSEP Project and published on February 5, 2014. The main environments used to perform the WAVSEP 2014 assessment, WAVSEP 1.5 and WIVET 3-rev148, were again made open source and published online alongside the comparative review (at http://code.google.com/p/wavsep/ and http://code.google.com/p/wivet/, respectively). WAVSEP 1.5 comes with a collection of over 1000 vulnerable test cases (1413 test cases to be more precise) for 6 different attack vectors.
Highlights
Various government institutes and agencies worldwide use the platform as an assessment platform for vulnerability scanners, often as the main one.
Generic Application-Level Vulnerability Detection Module: a collection of features that attempt to identify generic exposures in the application layer, without prior knowledge about the application and its structure, and while potentially overcoming barriers along the way. This module is the primary focus of this research.
The purpose of the accuracy assessment is to see how effective each scanner is in detecting a variety of vulnerabilities, and to see whether or not the detection logic "settles" for simple scenarios, or covers a collection of common and advanced scenarios.
Since the latest benchmark, many open source & commercial tools added new features and improved their detection accuracy.
Syhunt dramatically improved their WIVET score (came out FIRST, along with 3 others), and slightly improved other scores as well (LFI, etc). — WAVSEP 2014
Test Cases
The various scanners were tested against the following test cases (GET and POST attack vectors):
- 816 test cases that were vulnerable to Path Traversal / LFI attacks.
- 184 test cases that included Hidden, Obsolete and Backup files.
- 108 test cases that were vulnerable to Remote File Inclusion (XSS via RFI) attacks.
- 66 test cases that were vulnerable to Reflected Cross Site Scripting attacks.
- 80 test cases that contained Error Disclosing SQL Injection exposures.
- 60 test cases that were vulnerable to Phishing via Unvalidated Redirect.
- 46 test cases that contained Blind SQL Injection exposures.
- 10 test cases that were vulnerable to Time Based SQL Injection attacks.
Additionally, the various scanners were tested against 43 different categories of false positive, covering Unvalidated Redirect, Obsolete/Hidden/Backup files, Path Traversal / LFI, Remote File Inclusion, Reflected XSS vulnerabilities and SQL Injection vulnerabilities.
Results
WIVET - Crawling Coverage
In order to assess the scanners attack surface coverage, the assessment included tests that measure the efficiency of the scanner's automated crawling mechanism (input vector extraction) , and feature comparisons meant to assess its support for various technologies and its ability to handle different scan barriers.
This section of the benchmark also included the WIVET test (Web Input Vector Extractor Teaser v3-rev148), in which scanners were executed against a dedicated application that can assess their crawling mechanism efficiency in the aspect of input vector extraction.
If you use scanners mainly in a point-and-shoot scenario, and prefer as much automation as possible, a high WIVET score will be the second most important feature you should follow.
Organizations and QA/DEV departments really need a tool with a high score in this category – especially in 2014 — WAVSEP 2014
Syhunt Dynamic 5.0 was able to achieve 94% crawling coverage, showing that Syhunt Dynamic's enhanced crawler is well equipped for handling malformed HTML, many kinds of relative links, finding certain patterns in HTML comments, and for performing a variety of browser emulation tasks, such as JavaScript execution and DOM emulation, user interaction simulation, session and cookie handling, code analysis, and more.
SQL Injection (Blind and Time-Based)
One of the most famous exposures (and powerful attacks) and the most commonly implemented attack vector in web application scanners — WAVSEP 2014
Syhunt Dynamic achieved 100% detection of SQL Injection vulnerabilities (both Blind and Time Based).
XSS (Cross-Site Scripting)
This section deals with the detection accuracy of Reflected Cross Site Scripting, a very common exposure which is the 2nd most commonly implemented feature in web application vulnerability scanners. — WAVSEP 2014
Syhunt Dynamic achieved 100% detection of XSS vulnerabilities in the testing, with zero false positives thanks to its advanced filters.
XSS (Cross-Site Scripting) & SQL Injection - Stacked
Syhunt, Acunetix, AppScan, Netsparker and WebInspect top the WAVSEP 2014 benchmark charts with 100 percent SQL Injection and XSS detection rates.
Hidden, Obsolete & Backup Files
This test assessed the capabilities of scanners to locate backup files with non-executable extensions, compressed versions of files and directories that developers may have forgotten, sequential files or copies of files and directories that are remnants of various development tests, and additional hazards that may lead to source code configuration disclosure. — WAVSEP 2014
File Inclusion (RFI & LFI) - Stacked
Note: QualysGuard WAS is absent from the file inclusion chart because there were issues when trying to use its LFI/RFI plugins.
Phishing via Unvalidated Redirect
At the time that the comparison was performed Syhunt Dynamic could not perform checks for unvalidated redirect. A plugin for detecting unvalidated redirects has been added later.