Pilot Execution (15)

Description:

After pilot design and development (see Pilot Design and Development building block), the state will move on to pilot execution. This begins with the procurement of one or more vendors, which may include commercial account managers or mileage reporting technology vendors. Pilot execution also requires efforts to recruit pilot participants. For small pilots (50 to 200 participants), recruitment can be conducted by directly inviting stakeholders from various groups across the state. Larger pilots (more than 200 participants) may recruit from the general public, and these are often more time- and effort-intensive.

The next steps in pilot execution include onboarding and enrolling participants, conducting the live pilot, closing out the pilot, and analyzing pilot data. Pilot goals, high-level design, specification documents, and the communication plan inform pilot execution (see Develop and Execute Communication Plan and Collateral building block). Pilot execution includes overseeing vendors, providing pilot-level customer service if there is more than one vendor, and collecting vendor data and participant surveys.

Details:

Procurement of the vendors should follow state procurement policies/rules and use specification documents as a basis on which vendors bid. For recruitment efforts, a pilot web page can advertise and solicit pilot participation and can link to the participant enrollment form. The pilot web page can be advertised using some combination of social media, local media sources, department of motor vehicles renewal notices, or public information events. Recruitment for larger pilots may need to reach certain geographic or demographic quotas. In this case, the signup process should require participants to provide more detailed information, to ensure they meet pilot goals and criteria.

Following participant recruitment efforts, the pilot execution includes the following components:

  • Participant onboarding and enrollment. Get participants into the system and actively reporting mileage.
  • Vendor oversight. Make sure vendors deliver services as promised; review vendor communications with participants to ensure they align with pilot communications.
  • Customer service. Help participants choose between vendors and answer policy questions; this is provided in addition to vendor customer service that helps participants with vendor-provided systems.
  • Collection of vendor data. Secure transmittal of vendor data to the lead RUC agency.
  • Participant surveys. Solicit feedback about participants’ pilot experiences.
  • Pilot closeout. End the pilot, conduct final surveys and/or focus groups, and collect pilot hardware (if any).

Primary Uses:

Obtain system vendors for the pilot, recruit people to participate in the pilot, and complete the pilot.

Best Practices/Lessons Learned:

  • Consider holding a request for information (RFI) before a procurement to get industry feedback on specification documents. Documents may be updated based on industry feedback.
  • Use a best value procurement, awarding the bid to the best blend of appropriate technology and price. Advertise the request for proposal (RFP) widely and invite potential vendors to bid. More bids are generally better.
  • Require vendors to disclose all exceptions in their proposals and, ideally, provide equivalent alternative functionality because vendors may be allowed to take limited exceptions to the specifications.
  • Do not rush implementation or testing, even if implementation and testing timelines are longer than expected. Rushing can lead to lower-quality systems.
  • Ensure the pilot web page explains the RUC policy and pilot in an accurate, neutral way. Advocating for RUC at any point in the pilot can draw the neutrality of the pilot into question and limit the use of pilot results.
  • Consider using incentives like cash or gift cards to recruit larger numbers of public participants. These incentives can also be tied to the completion of certain activities (e.g., surveys, mileage reporting) to provide a stronger guarantee of obtaining pilot data and feedback.
  • Allow at least a month for enrollment before the start of live pilot data collection to ensure participants are actively reporting mileage data.
  • In large pilots, enroll participants in waves to avoid overwhelming customer service.
  • Keep participants engaged throughout the pilot by providing regular communications, such as driving statements and newsletters.
  • Depending on desired outcomes, pilot objectives, and pilot scope, consider a shorter pilot of 3 to 6 months; it may not be necessary to run a pilot for 12 months.
  • Have commercial account managers provide customer service for their own systems. If there is more than one commercial account manager, provide pilot-level customer service to help participants select a commercial account manager; otherwise, commercial account managers can provide all customer service. Keep in mind that participant policy questions often need to be answered by the state, even if the commercial account manager provides customer service.
  • Handle discontented participants with attention and care.
  • Consider communications activities performed in conjunction with and in support of a pilot, which are particularly important for recruitment.

State Government Context and Assumptions:

The lead RUC agency performs this task, with procurement officer support for obtaining the pilot vendors. Pilot specifications must be mature before procurement begins.