Pilot Design and Development (14)

Description:

If a state opts to conduct a pilot, clear objectives should be set. Primary pilot goals may include educating the public and soliciting their feedback about RUC, testing policy elements, or testing new technical or operational approaches. After objectives are set, a high-level design of the pilot should be created based on the policy vision, pilot goals, and prioritized mileage reporting methods. If commercial motor carriers are included in the RUC system plans, they also should be included in the pilot. Commercial motor carrier education, communications, and mileage reporting methods likely require separate planning from noncommercial vehicles given the different regulatory operating environments.

Next, develop pilot specification documents that are based on the high-level pilot design, and follow standard systems engineering practice. This includes developing multiple procedures and documents, such as a concept of operations, system requirement specification, an interface control document, and business rules. Finally, develop formal test plans to verify that the system operates according to the specifications. Tests generally include unit tests, integration tests, end-to-end (system acceptance) tests, and user interface testing via a small-scale operational trial before the pilot launch.

Details:

Set high-level goals and objectives for the pilot and then select key pilot parameters, including the following:

  • Number of commercial account managers or pilot operators
  • Number and composition of participants
  • Pilot duration
  • Special technical/operational approaches to test
  • Number of public surveys and/or focus groups to be conducted

After setting the goals and objectives, develop system specification documents. The first such document is the concept of operations, which uses nontechnical language to set the project context, identify stakeholders, and provide a conceptual design, high-level system architecture, and expected use cases.

When the concept of operations is complete, develop the following documents:

  • System requirement specification. This document refers to the technical requirements outlining what the system should do—not how the system should do it—and describes the minimum performance of the system.
  • Interface control document. This document details the interfaces between systems.
  • Business rules document. This document details the business rules for the system, including handling of money (real or simulated), customer service, rules for enrollment, and statements.
Finally, design tests should be performed in the following order:

  1. Unit tests test software systems or hardware in isolation. The system vendor (typically a technology vendor but occasionally the state in some circumstances) conducts these tests, then the state validates the tests by reviewing vendor records.
  2. Integration tests test whether any two systems are interfacing correctly. Vendors conduct these tests.
  3. End-to-end or system acceptance tests use a small number of test drivers to test the full system, from mileage reporting to statement generation. The lead RUC agency oversees these tests, with consultant support as needed.
  4. User interface/usability tests test the quality of the user interface using a small-scale operational trial. The test includes a larger number of test drivers from the normal participant pool. The lead RUC agency oversees this test, with consultant support as needed.

Primary Uses:

Design a pilot at both a high level and functional level to enable development of pilot specifications. Then, develop plans to test the system.

Best Practices/Lessons Learned:

  • Pilots generally push policy forward toward an operational system, and goals for the pilots may be selected to encourage such progress.
  • A key purpose of pilots is to educate the public and stakeholders on RUC. Pilots are an effective way to communicate with and educate the public on RUC through real-world experience.
  • Operational pilots and pilots focusing more on technology do not need as many participants as public education and policy-focused pilots. For operational and technology-focused pilots, 50 to 200 participants should be sufficient.
  • Public education and policy-focused pilots generally need larger public participation—between 1,000 and 5,000 participants—to be effective.
  • Choose the composition of participants for the feedback desired. For policy pilots, a mix of geographic and demographic groups, and maybe vehicle types, is desirable.
  • Documents from states with operational systems and earlier pilots can inform the development of specification documents.
  • The interface control document should be an open specification that is available to all. To guarantee an open system, this document should not use any proprietary technology.
  • The distinction between technical requirements and business rules is fuzzy. Technical requirements focus on technology, while business rules focus on business interactions; however, both make technical demands on system operations.
  • Requirements and business rules should be as simple and clear as possible to achieve the desired functionality.
  • Not all requirements can be tested, but many requirements can be checked instead using a certification process.
  • Sufficient time should be incorporated for testing and to address any issues that may arise. After all discovered errors are addressed, retesting is recommended.
  • Test the system once it is fully developed or when it is as close to final as possible.

State Government Context and Assumptions:

The task force, if it exists, can guide this task, with the lead RUC agency executing it. A pilot may not be necessary, but it could still prove beneficial to authorizing strategy. If a pilot is implemented, consideration should be given to whether and how to include commercial vehicles.