basics

Every new model launch comes with something like a report card: standardized problem sets in math, coding, and knowledge used to score and compare models.

What are these called?

Q: What are these called?

Benchmarks. Benchmarks are standard test sets for comparing models. Perennial controversies: test questions leaking into training data ('contamination') and the gap between scores and real-world feel.

Benchmarks ✓
Bookmarks
Watermarks

Benchmarks
Benchmarks are standard test sets for comparing models. Perennial controversies: test questions leaking into training data ('contamination') and the gap between scores and real-world feel.

▶ Quration AIQ

basics

What's the umbrella term?What are these pieces called?When people describe an AI model as having '70 billion parameters,' what are they counting?What is this phenomenon called?What was it called?What is this human-feedback training step called?What is this limit called?Llama, the flagship of 'open-weight' models anyone can download and run on their own servers, was released by which company?What do we call models that handle multiple forms of input and output?What are these 'reasoning models' doing during that pause?What's this approach called?What is it?

Quration — Quration AIQ