Federal Highways requested that we measure our performance on this project, including both monitoring our progress along the way (Are we on track?) and identifying the results of our efforts (How did it work out?). This post summarizes the performance measures that we selected.
Considerations
We wanted metrics that would tie to both outputs (did we do X?) and outcomes (did our work make a difference in Y?) for three categories:
- Tool implementation and deployment
- Capacity building and partnerships
- Technology transfer / research dissemination
We wanted to use metrics that lend themselves to a definitive and objective answer (i.e. yes, no, 1, 9, 33, etc)
We wanted to be able to conduct the evaluation without a significant amount of effort (since this request was added after we had done our budget)
Tool implementation and deployment
Outputs
Goal: Agency and project partners participate in all required calls/meetings.
Metric: Number of calls/meetings attended
Goal: Project deliverables are submitted to Volpe/FHWA on time and on schedule.
Metrics:
- Quarterly progress reports submitted by specified due date
- Final deliverables submitted by due date
Goal: Agency identifies desireable refinements (i.e., suggestions for future research) for tools created from the C10 project.
Metric: Documentation of desireable refinements within existing project deliverables [ included in final report ]
Goal: Agency supplies lessons learned from participating as a C10 grantee.
Metric: Documentation of lessons learned [ inclusion in final report ]
Outcomes
Goal: Travel demand model contains new sensititivities suitable for policy analysis.
Metric: Number of progress reports that document new variables / modeling options available [ 1+ ]
Goal: Methodologies, work processes, key decisions, problems encountered, & lessons learned are sufficiently well documented that peers can follow the work and repeat the results.
Metric: Number of issues and lessons documented in on-line tools [ 1+ ]
Capacity building and partnerships
Outputs
Goal: Agency practitioners (staff, contractors, consultants) and assigned partner staff are engaged with project and familiar with results.
Metric: Number of users of online collaboration tools [ 1+ / agency ]
Outcomes
Goal: Agency and partner staff acquire additional skills and expertise.
Metric: Number of progress reports that document new skills / expertise acquired [ 1+ ]
Goal: Improved work processes, data, analysis tools, and decision information are in use by our agencies.
Metric: Number of progress reports that document uptake of new processes, data, tools, methods [ 1+ ]
Technology transfer / research dissemination
Outputs
Goal: Project data and information is shared with the academic and practitioner communities.
Metrics:
- Number of presentations delivered (conferences, technical meetings, TRB) [ 1+ ]
- Number of papers/memos/articles written about the project experience [ 1+ ]
Outcomes
Goal: Peer agencies in the state/region expresss interest in or begin to deploy C10 tools.
Metric: Number of agencies that contact C10 team about the project and/or express plans to pursue implementation [ 1+ ]
What have been others’ experience in objectively measuring progress for this type of project?