In researching some testing solutions for my own work, I found an article in the IEEE library from a group of Microsoft researchers about automating the software testing process. (Godefroid, et al, 2008). They are taking the concepts of static analysis to the next level by researching and prototyping methods of generating harnesses for automated dynamic testing. They discuss four different projects for test automation, but the most interesting one for me in the article was a project called SAGE (scalable, automated, guided execution). The SAGE project is based on white box fuzz testing and is intended to help reduce the number of defects related to security. “Security vulnerabilities (like buffer overflows) are a class of dangerous software defects that can let an attacker cause unintended behavior in a software component by sending it particularly crafted inputs.” The solution is white box because the program under test is running under a debugger-like monitor. The monitor observes and catches runtime exceptions generated by the program as the testing suite is exercising it with a variety of dynamically generated invalid input data. The tester and monitor programs are able to record, pause and replay for engineers the history of events up to the exception causing the program to crash.

An early version of SAGE was able to find a defect in a Windows kernel-level library responsible for parsing animated cursor image files. The tool generated over 7,700 test cases based on sample input data from testers and exercised the library for a little more than seven hours before the defect was uncovered. After analysis of the SAGE data, a fix for the defect was released as a out-of-band security patch for Windows. The authors write, “SAGE is currently being used internally at Microsoft and has already found tens of previously unknown security-related bugs in various products.”

Reference

Godefroid, P., de Halleux, P., Levin, M. Y., Nori, A. V., Rajamani, S. K., Schulte, W., Tillmann, N. (2008). Automating Software Testing Using Program Analysis. IEEE Software. 0740-745908.