- What are other people doing and what is available?
- How do we change behavior?
- How do I know my program is working?
- What can help me plan my safety program?
Overview
Measures of effectiveness from research literature
A measure of effectiveness is a scale or standard that can assess change in the targeted behavior, knowledge, attitude, etc. An appropriate measure of effectiveness varies for each observable change and considerations for the priorities and resources of the organization must be made. The measurement tool utilized should be relevant, reliable or standardized, practical, and discerning. Below are some examples of measures of effectiveness that have been used in research literature.
Show
Measures of effectiveness by research program
A measure of effectiveness is a scale or standard that can assess change in the targeted behavior, knowledge, attitude, etc. An appropriate measure of effectiveness varies for each observable change and considerations for the priorities and resources of the organization must be made. The measurement tool utilized should be relevant, reliable or standardized, practical, and discerning. The evaluation design (how the measures are used) should also be appropriate for the conclusions that the organization wishes to make about an intervention. Below is a selection of research programs that may not address measures of effectiveness directly, but discuss dependent variables that represent measures that could potentially be used to assess the effectiveness of more general safety programs.
Show
Evaluation Design Overview
Description
Any safety program evaluation should begin by reflecting on what an ideal evaluation would entail, and then considering how best to meet that standard within a real-world context where time and resource constraints must be considered. In a hypothetical ideal evaluation, the definition of the problem will dictate the details of the program evaluation strategy and measures, as well as providing adequate pre- and post-data for the targeted behavior (e.g., speeding, seat belt use, etc.). However, in many real-world situations, for a variety of reasons, such specificity and data are not available. In practice, program evaluation will be based on one of three basic designs.
Filter By
- Pre-Program Only
- Post-Program Only
- Pre- / Post-Program
Measures of Effectiveness from Research Literature
Pre-Program Examples
Transit (pupil transportation services)
Industry: TransitEvaluation Design
Pre-Program only
Location
Virginia, United States of America
Metrics Used
Crash Reports, Survey/Test
SUMMARY: Trainee survey responses and data of past school bus crashes informed the performance and safety behaviors of a group of school bus drivers in Virginia.
Post-Program Examples
Random sample of Class A and Class B commercial freight drivers
Industry: TruckingEvaluation Design
Post-Program only
Location
Zhongshan, Jiangmen, and Zhaoqing, Guangdong Province, PRC
Metrics Used
Survey/Test
SUMMARY: After the cessation of a program focused on applying knowledge from training programs to the workplace, survey responses were used to measure the effectiveness of the program.
Local transit companies’ managers and executives
Industry: TransitEvaluation Design
Immediate Post-Program, Delayed Post-program
Location
Metropolitan area of Rio de Janeiro, Brazil
Metrics Used
Crash Reports, Interviews
SUMMARY: To measure the effectiveness of a twelve-hour training course, post-training visits, interviews, and crash records were analyzed. The training course aimed to increase knowledge of factors that contribute to traffic crashes as well as indicators of risk and problematic drivers.
Driver training at technical college
Industry: TruckingEvaluation Design
Immediate Post-Program (compared with previous cohorts)
Location
Eau Claire, Wisconsin, United States of America
Metrics Used
Incident Reports
SUMMARY: Researchers evaluated an eight-week truck driving training program at Chippewa Valley Technical College by studying 142 drivers before and after a virtual driving simulator was added to the course. Commercial driver's license scores, incident reports, and driver confidence in specific maneuvers were used as indicators of effectiveness.
The Federal Road Safety Commission (FRSC) public programs for commercial vehicle drivers
Industry: OtherEvaluation Design
Delayed Post-Program
Location
Urban centers in Southwestern Federal Republic of Nigeria
Metrics Used
Survey/Test
SUMMARY: Focus groups and surveys informed the effectiveness of a public enlightenment program to improve driving in Nigeria. Knowledge of poor driving behavior as well as the influence, compliance, and perception of the program's effectiveness was measured.
Wyoming Department of Transportation
Industry: Other, TruckingEvaluation Design
Immediate Post-Program, Delayed Post-Program
Location
State of Wyoming, United States of America
Metrics Used
Survey/Test
SUMMARY: For a group of professional snowplow truck drivers, the Wyoming Department of Transportation measured the effectiveness of their new connected vehicle (CV) training program, using previously collected data (demographics, driving experience, crash history, and experience with advanced driver assistance systems) and a questionnaire (knowledge of CV system and perceived success with it).
Pre- and Post-Program Examples
Dead Sea Works (DSW)
Industry: OtherEvaluation Design
Pre-Program, Immediate Post-Program
Location
Beersheba, Israel
Metrics Used
Crash Reports
SUMMARY: Dead Sea Works evaluated the effectiveness of a training initiative that included three workshops and individualized feedback on driving via pre-intervention and post-intervention driving behaviors (seatbelt use, turn signal use, and vehicle speed). Pre- and post-initiative crashes were also considered.
Public and private organizations with employees who drive as part of their jobs (excluded taxis and buses)
Industry: AllEvaluation Design
Pre-Program, Immediate Post-Program
Location
Great Britain, United Kingdom
Metrics Used
Interviews, Survey/Test
SUMMARY: Sixty companies in the United Kingdom adopted a CD-ROM training aimed toward improving awareness and knowledge of work-related road safety for their drivers. Interviews and surveys before and after determined effectiveness.
Four domestic trucking carriers
Industry: TruckingEvaluation Design
Pre-Program, Immediate Post-Program
Location
Pacific Northwest Region, United States of America
Metrics Used
Health Assessments, Survey/Test, Vehicle Metrics
SUMMARY: The effectiveness of a six-month training intervention focused on driver health (diet, exercise, weight) and driver performance (time driving over posted speed and hard braking) was measured using surveys before the program and after cessation. Vehicle monitoring and health assessments also informed effectiveness.
Sample of commercial motor vehicle (CMV) drivers
Industry: TruckingEvaluation Design
Pre-Program, Immediate Post-Program
Location
Michigan, United States of America
Metrics Used
Survey/Test
SUMMARY: A sample of truck drivers completed a two-step training program to identify training needs, and then a training program was designed to address those needs (driver performance of situational awareness, speed control, direction control, and attitude toward crashes). Participant surveys complemented pre-program and post-program observation.
Case studies of 12 companies from a variety of industry sectors
Industry: TruckingEvaluation Design
Pre-Progam, Immediate Post-Program
Location
Australia
Metrics Used
Crash Reports, Vehicle Metrics, Fuel Use, Injuries
SUMMARY: A variety of organizations that employ drivers of heavy vehicles participated in a safety program for the drivers. Crash rates and costs, lost-time injuries, insurance premiums, fuel consumption, kilometers traveled, number of trips, vehicle repairs, and maintenance costs were some topics that the program aimed to improve. Improvement was measured via surveys and crash rates.
Undisclosed large government agency in the Commonwealth of Australia
Industry: N/AEvaluation Design
Pre-program, Immediate Post-program
Location
Australia
Metrics Used
Self-Reported Data
SUMMARY: For a training for supervisors of work-related drivers, supervisors were pushed to improve mindfulness, pro-social motivation, role clarity, and leadership self-efficacy. The success of the program was determined using comparisons between pre-program and post-program survey results.
Undisclosed community care nursing agency at six of the service centers
Industry: OtherEvaluation Design
Pre-Program, Immediate Post-Program
Location
Australia
Metrics Used
Crash Reports, Survey/Test
SUMMARY: Previously collected information, such as crash history and perceptions of safety, in combination with a post-training survey provided evaluation information for a three-month training program on speeding. The training specifically focused on correcting beliefs that everyone speeds, that safe speeding is acceptable, and that speeding saves time.
One of Australia’s largest nonprofit organizations
Industry: OtherEvaluation Design
Pre-Program, Delayed Post-Program (3 & 6 months)
Location
Australia
Metrics Used
Vehicle Metrics
SUMMARY: Participants of a one-hour safety awareness session that utilized facilitated discussion and goal setting were evaluated based on their speeding habits on residential roads, highways, and to clients or their office.
Transit organization
Industry: TransitEvaluation Design
Pre-program, Immediate Post-Program
Location
Kosofe and mainland local government areas in Lagos, Nigeria
Metrics Used
Self-Reported Data, Survey/Test
SUMMARY: A three-phase training program was implemented in an effort to increase road safety knowledge, road safety behavior, and adherence to speed limits amongst minibus drivers. Results of a test measured road safety knowledge, and self-reported data, in turn, informed behaviors.
Seven bus companies
Industry: Transit, OtherEvaluation Design
Pre-program, Delayed Post-Program (one month)
Location
Gyeonggi, Incheon, Pusan, Seoul, and Yeosoo districts, Republic South Korea
Metrics Used
Vehicle Metrics
SUMMARY: In-vehicle monitoring data including speed, engine speed, brake signal, vehicle position, acceleration, and distance covered informed seven bus companies of the effectiveness of a week-long training program focused on dangerous driving behaviors.
British Petroleum’s Fabrics and Fibers business unit
Industry: TruckingEvaluation Design
Pre-Program, Immediate Post-Program, Delayed Post-Program (daily information collection)
Location
Georgia, United States of America
Metrics Used
Incident Reports, Peer-Reported Data, Self-Reported Data
SUMMARY: Several thousand trainees, including drivers, participated in a safety training program designed to decrease speeding and increase seat belt use, as well as encourage drivers to follow proper traffic flow in parking lots, alert non-drivers to the drivers' presence in high-traffic plant areas, and turn off unattended vehicles. Self- and peer-reported observations, incident rates, and daily scorecards informed the effectiveness of the program.
Emergency responders
Industry: OtherEvaluation Design
Pre-Program, Delayed Post-Program (6 months)
Location
Northern Sweden
Metrics Used
Peer-Reported Data, Self-Reported Data
SUMMARY: Following the implementation of a new eLearning tool, training was created for ambulance drivers. Effectiveness was measured using self-reported and peer-reported information as well as previously collected data.
Large building materials company
Industry: OtherEvaluation Design
Pre-Program, Delayed Post-Program (2 months)
Location
Eight cities in the Hellenic Republic, Greece
Metrics Used
Survey/Test
SUMMARY: In an effort to reduce fatigued driving, a targeted training program for drivers was implemented. Effectiveness was determined by survey responses, which reported on factors including but not limited to fatigue, behavioral beliefs about fatigue, risk comprehension about crashes, behavioral intentions in regard to fatigue, control beliefs about fatigue, descriptive norms about rest, past behaviors linked to tiredness, and behavior in response to fatigue.
Large transport company
Industry: TruckingEvaluation Design
Pre-Program, Immediate Post-Program
Location
Melbourne, Australia
Metrics Used
Crash Reports, Training Attendance Records, Hiring Records, Interviews
SUMMARY: A ten-hour training program sought to increase knowledge of defensive driving, collision avoidance, and vehicle safety issues. Previous crash reports, hiring records, training attendance records, and interviews determined whether the program was effective.
Truck fleet drivers
Industry: TruckingEvaluation Design
Early Pre-Program, Immediate Pre-Program, Immediate Post-Program, Delayed Post-Program
Location
Ten states within the Federative Republic of Brazil
Metrics Used
Fuel Use, Vehicle Metrics
SUMMARY: Data from driver trips, trailer weight, and fuel use informed a program aimed to increase safe driving behaviors (reduce speeding, hard braking, ratio of excessive speed time/total movement time), increase vehicle operational efficiency (reduce overrevving, idling, ratio time of movement/productive time), and reduce fuel consumption.
Four domestic middle-size logistics companies
Industry: TruckingEvaluation Design
Pre-Program, Delayed Post-Program (2 months & 4-5 months)
Location
Southern Finland area, Republic of Finland
Metrics Used
Self-Reported Data, Survey/Test
SUMMARY: Four companies in Finland collected and used self-reported data on driver sleepiness, prior sleep, and the use of sleepiness countermeasures in combination with previous hourly logs to measure the effectiveness of a three-and-a-half-hour educational intervention for a group of 53 truck drivers.
Förbom Yhtiöt (Finnish transit)
Industry: TransitEvaluation Design
Pre-program, Delayed Post-Program (1 month & 2 years)
Location
Republic of Finland
Metrics Used
Crash Reports, Self-Reported Data
SUMMARY: Finland’s Transit organization implemented a 20-hour training program designed to increase defensive driving, most notably maintaining a “safety margin” and recognizing risk factors/attitudes about other drivers, for their 86 drivers. Self-reported and crash data two years before and two years after the program informed the effectiveness of the program.
Filter By
- Pre-Program Only
- Post-Program Only
- Pre- / Post-Program
Measures of Effectiveness by Research Program
Pre-Program Examples
Stages of Change/Transtheoretical Model of Behavior Change
Banks, T. D. (2008). An investigation into how work-related road safety can be enhanced (Doctoral dissertation). Queensland University of Technology. Banks, T. D., Davey, J. D., Biggs, H. C., & King, M. J. (2008). Stages of change in the Australian workplace and its application to driver education [Paper presentation]. 3rd International Driver Behavior and Training Conference, Dublin, Ireland. Banks, T. D., Davey, J. D., & Brownlow, D. M. (2006). Driver education and fleet safety climate in an emergency service fleet. Journal of Occupational Health and Safety, Australia and New Zealand, 22(4), 341–350.Evaluation Design
Pre-Program only
Studies
Three studies
SUMMARY: The three studies focused on readiness for change and safety climate amongst employees who drive on the job. The stages of change were described using self-reported data and interviews.
Post-Program Examples
Theory of Planned Behavior, by Wills
Wills, A. R., Biggs, H. C., & Watson, B. (2005). Analysis of a safety climate measure for occupational vehicle drivers and implications for safer workplaces. The Australian Journal of Rehabilitation Counselling, 11(1), 8–21. Wills, A. R., Watson, B. C., & Biggs, H. C. (2004). The relative influence of fleet safety climate on work-related driver safety. In J. Doak (Ed.), Proceedings of the Road Safety Research, Policing and Education Conference (pp. 1–12). Office of Road Safety. Wills, A. R., Watson, B., & Biggs, H. C. (2006). Comparing safety climate factors as predictors of work-related driving behavior. Journal of Safety Research, 37(4), 375–383. Wills, A. R., Watson, B., & Biggs, H. C. (2009). An exploratory investigation into safety climate and work-related driving. Work, 32(1), 81–94.Evaluation Design
Post-Program only
Studies
Four studies
SUMMARY: The responses to a 30-item questionnaire given by driving employees informed the role of safety culture at various organizations.
Pre- and Post-Program Examples
Cross-Sectional Correlation—No Theory
Öz, B., & Lajunen, T. (2008). Effects of organisational safety culture on driver behaviours and accident involvement amongst professional drivers. In Driver Behaviour and Training: Volume III (pp. 143–153). Ashgate. Öz, B., Özkan, T., & Lajunen, T. (2013). An investigation of professional drivers: Organizational safety climate, driver behaviours and performance. Transportation Research Part F: Traffic Psychology and Behaviour, 16, 81–91. Özkan, T., & Lajunen, T. (2005). A new addition to DBQ: Positive driver behaviours scale. Transportation Research Part F: Traffic Psychology and Behaviour, 8(4–5), 355–368.Evaluation Design
ANY
Studies
Three studies
SUMMARY: Responses to survey items were used to validate a series of measures. These measures of effectiveness provide employer insight into driver behavior (violations, driver errors, driver offenses, and crashes) and safety culture.
Transtheoretical Model of Behavior Change
Olson, R., Anger, W., Wlliot, D., Wipfli, B., & Gray, M. (2009a). A new health promotion model for lone workers: Results of the Safety & Health Involvement for Truckers (SHIFT) pilot study. Journal of Occupational and Environmental Medicine, 51(11), 1233–1246. Olson, R., Anger, W. K., Elliot, D. L., Schmidt, S., & Gray, M. (2009b). Competition, Computers, and Coaching: Three C’s for More Effective Health and Safety Promotion with Truck Drivers. Presented at 88th Annual Meeting of the Transportation Research Board, Washington, DC. Olson, R., Wipfli, B., Thompson, S. V., Elliot, D. L., Anger, W. K., Bodner, T., Hammer, L. B., & Perrin, N. A. (2016). Weight control intervention for truck drivers: The SHIFT randomized controlled trial, United States. American Journal of Public Health, 106, 1698–1706.Evaluation Design
Immediate Post-Program Pre-Program
Studies
Three studies
SUMMARY: The frequency at which company truck drivers engage in non-compliance (seatbelts, hard braking, hours of service) behaviors were measured via archival and self-reported data. The effectiveness of the six-month program was determined using a pre-and post-program design.
Theory of Planned Behavior, Behavior Modification, and Feedback Theory
Lewis, I., & Newnam, S. (2011). The development of an intervention to improve the safety of community care nurses while driving and a qualitative investigation of its preliminary effects. Safety Science, 49(10), 1321–1330. Newnam, S., Greenslade, J., Newton, C., & Watson, B. (2011). Safety in occupational driving: Development of a driver behavior scale for the workplace context. Applied Psychology, 60(4), 576–599. Newnam, S., Griffin, M. A., & Mason, C. (2008). Safety in work vehicles: A multilevel study linking safety values and individual predictors to work-related driving crashes. Journal of Applied Psychology, 93(3), 632–644. Newnam, S., Lewis, I., & Watson, B. (2012). Occupational driver safety: Conceptualising a leadership-based intervention to improve safe driving performance. Accident Analysis & Prevention, 45, 29–38. https://doi.org/10.1016/j.aap.2011.11.003 Newnam, S., & Oxley, J. (2016). A program in safety management for the occupational driver: Conceptual development and implementation case study. Safety Science, 84, 238–244. Newnam, S., Tay, R., & Mason, C. (2006). Using psychological frameworks to inform the evaluation of fleet safety initiatives. Safety Science, 44(9), 809–820. Newnam, S., & VonSchuckmann, C. (2012). Identifying an appropriate driving behaviour scale for the occupational driving context: The DBQ vs. the ODBQ. Safety Science, 50(5), 1268–1274. Newnam, S., & Watson, B. C. (2009). A participative education program to reduce speeding in a group of work-related drivers. In R. Grzebieta & D. McTiernan (Eds.), Proceedings of the 2009 Australasian Road Safety Research, Policing and Education and the 2009 Intelligent Speed Adaption (ISA) Conference. Roads and Traffic Authority of New South Wales, Australia, pp. 527–534. Newnam, S., Watson, B., & Murray, W. (2004). Factors predicting intentions to speed in a work and personal vehicle. Transportation Research Part F: Traffic Psychology and Behaviour, 7(4–5), 287–300.Evaluation Design
Delayed Post-Program Immediate Post-Program Pre-Program
Studies
Nine studies
SUMMARY: Fleet drivers and managers participated in studies aimed at understanding attitudes, intentions to speed, and actual speeding in the presence of a participative education intervention. Archival and self-reported data over time informed the effectiveness of the intervention.
Goal Setting and Feedback Theory
Ludwig, T. D., & Geller, E. S. (1991). Improving the driving practices of pizza deliverers: Response generalization and moderating effects of driving history. Journal of Applied Behavior Analysis, 24(1), 31–44. https://doi.org/10.1901/jaba.1991.24-31 Ludwig, T. D., & Geller, E. S. (2000). Intervening to improve the safety of delivery drivers: A systematic behavioral approach. Journal of Organizational Behavior Management, 19(4). https://doi.org/10.1300/J075v19n04_01 Geller, E. S., & Hahn, H. A. (1984). Promoting safety belt use at industrial sites: An effective program for blue collar employees. Professional Psychology: Research and Practice, 15(4), 553.Evaluation Design
Delayed Post-Program Immediate Post-Program Pre-Program
Studies
Three studies
SUMMARY: Delivery vehicle drivers (industrial and pizza) participated in studies focused on incentives, "awareness sessions," and seatbelt use. Data from before and after the program was utilized to determine effectiveness.
Theory of Planned Behavior
Douglas, M. A., & Swartz, S. M. (2009). A multi‐dimensional construct of commercial motor vehicle operators’ attitudes toward safety regulations. The International Journal of Logistics Management, 20(2), 278–293. Swartz, S. M., & Douglas, M. A. (2009). The independence of independents: Influences on commercial driver intentions to commit unsafe acts. Transportation Journal, 48(1), 23–41.Evaluation Design
ANY
Studies
Two studies
SUMMARY: After creating and validating a 13-item survey measure of safety climate, the results were used to inform the interaction of safety climate and driver intentions at an organization. It can also be used for future measures of effectiveness.
Planned Behavior and Health Belief Model
Adamos, G., & Nathanail, E. (2015). How to train safe drivers: Setting up and evaluating a fatigue training program. Transport and Telecommunication Journal, 16(1), 9–20. Adamos, G., & Nathanail, E. (2017). A before-after study of the impact of a road safety training program on professional drivers. International Journal of Transportation, 5(1), 15–34. Adamos, G., Nathanail, E., & Kapetanopoulou, P. (2013). Do road safety communication campaigns work? How to assess the impact of a national fatigue campaign on driving behavior. Transportation Research Record, 2364(1), 62–70. Adamos, G., Nathanail, E., & Kapetanopoulou, P. (2014). Cluster modeling of driving behavior under fatigue. Transportation Research Record, 2465(1), 24–32. Nathanail, E., & Adamos, G. (2013). Road safety communication campaigns: Research designs and behavioral modeling. Transportation Research Part F: Traffic Psychology and Behaviour, 18, 107–122.Evaluation Design
Post-Program (timing varies) Pre-Program
Studies
Five studies
SUMMARY: Change in knowledge, attitudes, and compliance were measured with previously collected archival and self-reported data. The studies focused on the communication of campaigns for professional and non-professional drivers.