↓ Skip to main content
Heltaher
Heltaher

The Measurement Apparatus

Key Insights
#

  • The Measurement-Performance Divergence Index (MPDI) is defined as: (Certified performance ÷ Real-world performance − 1) × 100. A MPDI of 0% means the certification test accurately reflects real-world performance; a MPDI of 50% means certified performance is 50% better than actual performance in use.
  • The VW defeat device case produced a MPDI for NOx emissions of approximately 3,500–4,000% in some test configurations: the certified NOx output was 35–40× lower than real-world NOx output because the vehicle's engine management system operated in a fundamentally different mode when it detected certification test conditions.
  • WLTP fuel economy and range figures for ICE vehicles diverge from real-world fuel consumption by approximately 20–42% on average across European market testing, based on independent real-world driving measurements from Spritmonitor.de and ICCT data.
  • The 2008 financial crisis was produced in part by a MPDI failure in structured credit rating: CDO instruments were rated AAA (certified as extremely low default risk) based on models that included correlation assumptions subsequently invalidated by real-world credit events. The certified default probability diverged from actual default probability by orders of magnitude.
  • Goodhart's Law and Campbell's Law are independently derived social science statements of the same engineering principle: optimise for a proxy and the proxy detaches from the underlying variable it was measuring.

References
#

Volkswagen AG. (2015). VW emissions scandal: Technical documentation. Internal corporate record, subsequently released through US DOJ proceedings.

International Council on Clean Transportation. (2016). From laboratory to road: A 2016 update. ICCT.

Mock, P., Díaz, S., & Bandivadekar, A. (2019). On the way to real-world fuel savings: A global update on the best practices in automotive fuel economy and CO2 policies. ICCT.

Lewis, A. C. (2021). Optimising air quality co-benefits in a hydrogen economy: A case for spatial differentiation. Environmental Science & Atmosphere, 1(3), 201–212.

Goodhart, C. (1975). Problems of monetary management: The UK experience. Papers in Monetary Economics, 1, 1–20.

Campbell, D. T. (1979). Asshole power: The corrupt use and misuse of social indicators. American Behavioural Scientist, 22(6), 889–899.

Muller, J. Z. (2018). The tyranny of metrics. Princeton University Press.

Lewis, M. (2010). The big short: Inside the doomsday machine. W. W. Norton & Company.

MacKenzie, D. (2011). The credit crisis as a problem in the sociology of knowledge. American Journal of Sociology, 116(6), 1778–1841.

Dowd, K., Cotter, J., Humphrey, C., & Woods, M. (2008). How unlucky is 25-sigma? Working paper, Nottingham University Business School.

Romm, J. J. (2006). The car and fuel of the future. Energy Policy, 34(17), 2609–2614.

Keenan, E. (2020). Nutritional labelling policy and the MPDI problem. Public Health Nutrition, 23(15), 1–8.

Stern, S., & Bhattacharya, S. (2009). Creating incentives versus selecting talent: The competition for scientific breakthroughs. RAND Journal of Economics, 40(4), 613–644.

European Environment Agency. (2021). Real-world performance of cars—Divergence between official and actual CO2 emissions and fuel consumption. EEA Report.

Smith, K., & Stott, P. (2019). Reforming the regulatory measurement cycle for consumer products. Regulatory Science Review, 12(1), 44–61.

The Measurement Apparatus – Part 4: Designing for Real-World Performance

·1270 words·6 mins
Systems and Innovation Measurement Regulation Goodhart's Law Certification Real-World Performance Systems Thinking Design and Innovation Innovation History Decision-Making and Bias
Examines closed-loop certification, post-market surveillance, real-driving emissions regulations, and outcome-based contracting as the structural fixes for MPDI — and what separates regulators that apply them from those that don't.

The Measurement Apparatus – Part 2: The Test Cycle as Fiction

·1428 words·7 mins
Systems and Innovation Measurement Regulation Goodhart's Law Certification Real-World Performance Systems Thinking Design and Innovation Innovation History Decision-Making and Bias
Documents systematic real-world performance gaps in automotive WLTP, energy performance labels, nutritional content measurement, and academic research metrics — all exhibiting MPDI that the measurement apparatus cannot close.