Lesson
Establish a strong acceptance testing program, with independent code reviews, for ITS projects with software product deliveries.
A Washington State Department of Transportation implementation of a regional ATMS.
12/1/2002
Seattle,Washington,United States
Background (Show)
Lesson Learned
Develop a strong acceptance testing program as part of large-scale software product deliveries, with independent code reviews performed to verify functionality. A testing program featuring software code reviews by a qualified independent contractor will provide reassurances that existing or newly developed code meets its functional design requirements, and will also minimize the possibility of unrealistic product expectations driven by untested claims. The testing program should include documentation sufficient to enable the customer to understand module functionality, inputs and outputs, and installation processes, as well as the ability to perform at least top-level maintenance. Specific software deliverables should be identified in the task plan.
The NSATMS project's testing process was hampered by several issues:
The culmination of the project was what the client believed to be an unsuccessful delivery of the complete, working arterial traffic management system that it had expected to receive. The issue of whether, and to what extent, complete and functioning products were delivered under this contract became a source of discussion during the evaluation process and suggested a difference of opinion regarding the nature of the product that was delivered. If this difference had been resolved at the outset of the project, and a testing program had been put in place, the opportunity for greater customer satisfaction and project success would have been higher.
The NSATMS project's testing process was hampered by several issues:
- There was a disconnect between the client’s and the contractor’s definitions of a delivered, tested product. While the contractor noted that software modules were delivered, installed, and tested for functionality, the client believed that the tests were not comprehensive and were not reinforced with support tools, documentation, or training sufficient to enable the client staff to perform its own tests and verification. This difference of viewpoint suggests that a mutually agreed-upon software testing and acceptance program, with accompanying user operations manuals and system support documentation, would have enhanced the likelihood of customer satisfaction and overall project success.
- The testing processes, combined with a misperception of the software development aspects of the project, contributed to a system that was not fully operational from the client perspective. As noted in a related lesson, there was a difference of perception regarding the extent to which this project was primarily a modified turnkey installation project, rather than a software development project. A clearer understanding of the software development tasks of this effort would likely have highlighted the importance of mutually agreed-upon testing and validation procedures.
The culmination of the project was what the client believed to be an unsuccessful delivery of the complete, working arterial traffic management system that it had expected to receive. The issue of whether, and to what extent, complete and functioning products were delivered under this contract became a source of discussion during the evaluation process and suggested a difference of opinion regarding the nature of the product that was delivered. If this difference had been resolved at the outset of the project, and a testing program had been put in place, the opportunity for greater customer satisfaction and project success would have been higher.
Application Areas
None defined
States
Countries
Focus Areas
None defined
Goal Areas
None defined
Keywords
None defined
Lesson ID: 2005-00111
Lesson Comments
No comments posted to date