Secret Dates in System Prompts Undermine Language Model Evaluation

본문 미리보기

Without the ability to benchmark Large Language Models (LLMs), it is difficult for consumers and businesses to understand what progress a model has made over recent versions, and how it stands up to its competitors: Since LLMs are non-deterministic (i.e., they will not always produce consistent outputs given the same inputs), evaluating them is tricky. Even when researchers use identical prompts, model settings and benchmark datasets, seemingly minor differences in the execution environment can…