Training large language models is expensive – and evaluating them is, too. Many important skills, like solving a math problem or writing a short piece of code, are tested by asking the model to write out the full answer. That takes time and compute, so these “full-answer” tests are usually executed only rarely during training. In our EMNLP 2025 paper, we explore a simple idea: using much cheaper multiple-choice tests as an early, approximate indicator of model progress during (pre)training, providing a rough sense of whether training is proceeding as expected without relying on expensive generative free-text evaluations.
To do this, we take well-known benchmarks where models normally have to generate an answer and turn them into multiple-choice versions. For each question, we keep the correct answer and add several wrong but plausible alternatives (“distractors”), and we also use an even lighter variant where the model simply scores the correct answer instead of generating anything. We then test how well these reformulated tasks track the originals by measuring the performance correlation between both versions across 8 base language models of different sizes and 4 key capability areas: mathematical reasoning, code generation, factual knowledge, and reading comprehension.
The graphic below illustrates this with a toy example: the same question is answered in three ways
- just scoring the correct answer, token by token
- picking from multiple-choice options
- full text generation
The table highlights how the cheaper formats need far fewer model steps to get to a result.
The outcome is clear: using the multiple choice variant as an indicator for model performance works. These lower-cost tests closely mirror the improvements and setbacks seen in the original generative benchmarks and reliably show which models perform better than others. At the same time, they are dramatically faster – on average up to 35×, and in some code-heavy settings up to 176×. This makes it practical to evaluate core capabilities much more often during training and reserve the slow, full-answer benchmarks for occasional spot checks.
To make this usable in practice, we provide ready-made multiple-choice versions of several popular benchmarks in our EvalShortcut repository. You don’t have to build your own distractors or transformation scripts – you can plug these datasets into your existing evaluation setup and immediately benefit from faster, cheaper monitoring during model training.
Author: Dr. Darina Gold from Fraunhofer IIS
