EX-ANTE EVALUATION OF APPLIED RESEARCH AND INNOVATION PROJECTS: A SELF-AUDIT OF A GATED PROCEDURE WITH REDUCED-COMPENSABILITY AGGREGATION
DOI:
https://doi.org/10.62405/osiconf.2026.15Keywords:
ex-ante evaluation; composite indicator; compensability of aggregation; reference class; language-model sycophancy.Abstract
The aim is to document the design of a gated ex-ante evaluation procedure for applied research and innovation projects and to subject it to a self-audit rather than assert its advantages. Methods: analytical stress-testing of the aggregation and score-correction rules across the full admissible range of factor values; a run on a calibration set of 22 historical cases; and comparison of the claimed safeguard against language-model bias with existing benchmarks. Results: two design defects were found. The weighted geometric mean reduces but does not eliminate compensability – a factor at the normalisation floor of 0.02 with weight 0.20 still yields an index of 0.457 against a threshold of 0.35. The evidence-class correction inverts: below a confidence weight of 0.143 the whole attainable range lies above the threshold, so no idea can be rejected. Conclusions: corrections to both rules, a blind out-of-sample protocol and a preregistered experiment are proposed.
References
CoARA. (2022). Agreement on reforming research assessment. Coalition for Advancing Research Assessment. https://www.coara.org/agreement/
Cooper, R. G. (1990). Stage-gate systems: A new tool for managing new products. Business Horizons, 33(3), 44–54. https://doi.org/10.1016/0007-6813(90)90040-I
Flyvbjerg, B. (2006). From Nobel Prize to project management: Getting risks right. Project Management Journal, 37(3), 5–15. https://doi.org/10.1177/875697280603700302
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a
Ho, S.-T., Liu, M., Nghiem, H., & Huang, F. (2026). SoundnessBench: Can your AI scientist really tell good research ideas from bad ones? arXiv. https://doi.org/10.48550/arXiv.2605.30329
Kahneman, D., & Lovallo, D. (1993). Timid choices and bold forecasts: A cognitive perspective on risk taking. Management Science, 39(1), 17–31. https://doi.org/10.1287/mnsc.39.1.17
Nagy, B., Farmer, J. D., Bui, Q. M., & Trancik, J. E. (2013). Statistical basis for predicting technological progress. PLoS ONE, 8(2), e52669. https://doi.org/10.1371/journal.pone.0052669
Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffman, A., & Giovannini, E. (2008). Handbook on constructing composite indicators: Methodology and user guide. OECD Publishing. https://doi.org/10.1787/9789264043466-en
NASA. (2020). Technology readiness assessment best practices guide (NASA/SP-20205003605). National Aeronautics and Space Administration. https://ntrs.nasa.gov/citations/20205003605
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., … Perez, E. (2023). Towards understanding sycophancy in language models. arXiv. https://doi.org/10.48550/arXiv.2310.13548
Strathern, M. (1997). "Improving ratings": Audit in the British University system. European Review, 5(3), 305–321. https://doi.org/10.1002/(SICI)1234-981X(199707)5:3<305::AID-EURO184>3.0.CO;2-4
Wright, T. P. (1936). Factors affecting the cost of airplanes. Journal of the Aeronautical Sciences, 3(4), 122–128. https://doi.org/10.2514/8.155
Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3
