Establish a strong acceptance testing program, with independent code reviews, for ITS projects with software product deliveries.
A Washington State Department of Transportation implementation of a regional ATMS.
The NSATMS project's testing process was hampered by several issues:
- There was a disconnect between the client’s and the contractor’s definitions of a delivered, tested product. While the contractor noted that software modules were delivered, installed, and tested for functionality, the client believed that the tests were not comprehensive and were not reinforced with support tools, documentation, or training sufficient to enable the client staff to perform its own tests and verification. This difference of viewpoint suggests that a mutually agreed-upon software testing and acceptance program, with accompanying user operations manuals and system support documentation, would have enhanced the likelihood of customer satisfaction and overall project success.
- The testing processes, combined with a misperception of the software development aspects of the project, contributed to a system that was not fully operational from the client perspective. As noted in a related lesson, there was a difference of perception regarding the extent to which this project was primarily a modified turnkey installation project, rather than a software development project. A clearer understanding of the software development tasks of this effort would likely have highlighted the importance of mutually agreed-upon testing and validation procedures.
The culmination of the project was what the client believed to be an unsuccessful delivery of the complete, working arterial traffic management system that it had expected to receive. The issue of whether, and to what extent, complete and functioning products were delivered under this contract became a source of discussion during the evaluation process and suggested a difference of opinion regarding the nature of the product that was delivered. If this difference had been resolved at the outset of the project, and a testing program had been put in place, the opportunity for greater customer satisfaction and project success would have been higher.
Author: John M. Ishimaru and Mark E. Hallenbeck
Published By: Washington State Transportation Center (TRAC)
Source Date: 12/1/2002
EDL Number: 13818URL: http://ntl.bts.gov/lib//jpodocs/repts_te//13818.html
Average User Rating