Provide for large sample sizes when conducting before/after data collection efforts, to avoid impacting the ability to reveal statistically significant differences during the evaluation's statistical analysis.
Orlando, Florida's experience with a Field Operational Test (FOT) on using a single smart card for transportation payments at facilities operated by multiple regional agencies.
- Avoid delays in collecting data, so that data collection problems can be detected early. Collect any data being gathered with unattended equipment frequently, to avoid long periods of data loss in the event of equipment problems. LYNX attempted to gather accumulated Automatic Passenger Counter (APC) data from the equipped buses near the end of the after data collection period (APC data was used to measure before and after average boarding times). Unfortunately, only at that point did LYNX become aware that door sensor malfunctions had prevented APC data collection throughout much of the after data collection period. If the data had been retrieved more frequently (e.g., weekly) the door sensor problems could have been identified and resolved quickly, providing much more data.
The ORANGES evaluation suggests the following guidance for similar future initiatives:
- Plan to collect a substantial amount of before and after data so that the statistical evaluation of the data is more likely to reveal statistically significant differences. The number of observations will depend on the experimental design, the predicted magnitude of the effect, and the expected variance of the observations.
- Plan to include someone with expertise in experimental design and statistics to help design the experiment, create the data collection and analysis plan, and analyze and interpret the data.
- Include contingency plans that allow for supplementary data collection, if deemed warranted based on the initial statistical assessment of the initially collected data.
Developing a regional smart card payment system is related to the ITS Goal of improving customer satisfaction, through making payments for multimodal travel easier by establishing a regional payment method. The limitations in the ability to draw conclusions on statistically significant differences did not affect customer satisfaction.
The overall project was completed behind schedule. If the project had not been well behind schedule for other reasons, there would have been more flexibility to consider taking time for supplementary data collection.
The participating cardholders generally expressed a positive opinion about the technology, with concerns focusing primarily on the limited scale of deployment. Data collection issues were not a factor that limited the scale of deployment. In fact, a more extensive deployment would have inherently created additional data collection opportunities.
The ORANGES experience has demonstrated that there are a number of important issues in experimental design and data collection to maximize the chances of detecting significant differences associated with the before and after conditions study conditions. These issues include avoiding delays in collecting data, collecting a substantial amount of before and after data, and planning to include someone with expertise in experimental design and statistics. It is also important to include contingency plans that allow for supplementary data collection, if there are problems identified with the primary data collection effort.
Author: Moniz, Leisa M. (Volpe) et al.
Published By: U.S. DOT Federal Transit Administration
Source Date: 8/1/2004
EDL Number: 14268URL: http://ntl.bts.gov/lib//jpodocs/repts_te/14268.htm
U.S.DOT/RITA/John A. Volpe National Transportation Systems Center
Central Florida Regional Transportation Authority (LYNX)
Volpe National Transportation Systems Center
Average User Rating
Show the V
smart cards, electronic fare payment, SmartCard, smart card, SmartCards, Smartcard