Incorporate mechanisms for capturing user feedback for system evaluation, including the ability to intercept incoming 511 calls for survey or focus group recruitment.
An Arizona Department of Transportation experience in 511 implementation.
- Build user input tools into the system instead of adding them later. Incorporating tools to capture user input from the start is usually more cost-effective than adding them when the system is up and running. Moreover, having the input mechanisms built into the system early allows the operator to easily engage and disengage data collection, including the option of continuously collecting input.
- Use caller comments to help debug and refine the system. The caller comment feature has proven to be a very useful source of user input. It was especially useful in the first few months following the roll-out of the enhanced system when intensive debugging and refinement of the user interface was occurring. Caller comments were very useful -- both in demonstrating callers sensitivity to what many of them considered to be an inadequately performing user interface and in identifying and correcting specific malfunctions. Utilization of the caller comment featured varied over the first six months of post-enhanced operation from about 0.2% to 0.5% of all 511 calls, with the number of comments ranging from 37 to 373 per month.
- Use a live intercept survey method to get a higher response rate and a less biased sample of users. Initially ADOT objected to intercepting 511 callers for survey recruitment. They felt that callers would view a brief recorded message inviting them to leave their contact information, or an option to complete an automated survey on the phone, as less disruptive than a live operator intercept of their call. The evaluation team argued that respondents are more likely to agree when invited to participate by a human recruiter. A higher overall response rate in turn reduces sample bias, that is, it mitigates the tendency for only strongly positive and negative callers to participate. ADOT agreed and the live intercept approach was successfully implemented. Approximately 31% of intercepted callers agreed to be surveyed and approximately 71% followed through, yielding 411 surveys and an overall response rate of 22% (that is, 22% of those intercepted completed a survey—a very respectable participation rate.) No complaints were received from intercepted callers regarding the method of recruitment of survey completion.
User feedback is essential in developing a 511 system that is responsive to user needs and preferences. Effectively capturing user feedback is important both for debugging new or enhanced systems and for on-going monitoring of system effectiveness and marketing efforts. Building mechanisms to collect user input into 511 systems as they are being developed is more cost-effective than adding mechanisms later. Two key mechanisms that should be included are a caller comment feature allowing callers to leave voicemail comments and the ability for operators to intercept incoming 511 calls for live recruitment of user survey participants.
Author: Battelle and the University of Arizona
Published By: Federal Highway Administration, U.S. DOT
Prepared by Battelle and the University of Arizona for the U.S. DOT
Source Date: 30 September 2005
EDL Number: 14248
Other Reference Number: Report No. FHWA-JPO-06-013URL: http://ntl.bts.gov/lib//jpodocs/repts_te//14248.htm
Average User Rating