Project Description

Swiss Economics was commissioned to validate the AI-generated paper rankings of Kurate.org, an automated ranking system that uses large language models (LLMs) to assess the potential impact of academic preprints.

Our validation report compares several Kurate ranking methods against observed ratings by human experts in the context of the ICLR 2026 review process. The analysis shows that Kurate produces paper rankings that align closely with expert assessments, and in some cases surpasses the level of agreement observed between individual human experts and the decisions of the ICLR program committee. Results are robust across research subfields and replicated across different samples.

The findings support Kurate as a credible and cost-effective tool for automated paper screening and raise interesting questions about where AI can meaningfully complement the academic review process.

Methods

• Rank correlation analysis
• Documentation for Kurate.org

Publications

• Validation report