# Nano LM Index: protocol (V1)

The Nano LM Index measures open-weight language models with at most 600M parameters on a frozen suite of 12 benchmarks in six equally weighted categories. Every entry is run with the same prompts, extractors and scoring, and with its own published decoding settings. No setting is tuned per model. Scores are chance-corrected: 0 means no better than a no-skill strategy (the best constant answer, or uniform guessing), and 100 means perfect.

This document is the standard for V1. Anything that changes a score belongs in a new edition (see [Editions](#10-editions)) and is never applied silently.

## 1. Principles

1. **One protocol for all models.** If a setting would have to differ between models, the model's own artefacts decide it, never the maintainers. Those artefacts are its chat template, its context window and its published decoding settings.
2. **No selection on outcomes.** Never pick a model's better mode, prompt, run or subset after seeing scores.
3. **Deterministic scoring.** No LLM judges. Every benchmark is scored by code that gives the same verdict every time.
4. **Lenient on form, strict on content.** Any reasonable way of writing an answer is accepted (code fences, tags, `Answer:` lines, `\boxed{}`), but the answer itself must be exactly right.
5. **Unanswered is wrong.** A response that is unparseable, truncated, errored or does not fit the context window scores 0. It is counted and reported, never dropped.
6. **Everything is pinned.** Dataset revisions, model revisions, engine versions and chance levels are fixed, and raw outputs are kept, so a scoring fix is a rescore, never a rerun.

## 2. Eligibility and entries

**A model is eligible if all of the following hold:**

- **Open weights.** The weights are public on the Hugging Face Hub, under a licence that permits evaluation and publishing the results.
- **Chat or instruct variant.** It ships an official chat template. Base-only models cannot follow the shared protocol.
- **At most 600,000,000 parameters.** Counted from the unique tensors in the safetensors files at the pinned revision, not from the model card. Tied input and output embeddings count once; untied embeddings count twice, because they are separate weights.

**An entry is a model at a pinned revision in one mode.** The mode follows from the model's chat template, not from a maintainer's choice:

| Model type | How it is identified | Entries |
|---|---|---|
| Non-reasoning | No reasoning output and no switch in the template | `no-think` |
| Reasoning-only | Always reasons; the template has no off switch | `think` |
| Hybrid | The template has an official switch (e.g. Qwen3 `enable_thinking`) | Both `think` and `no-think`, as two entries |

- Both entries of a hybrid model are ranked on the same board. No "best mode" score is published for a model.
- Modes are switched only through the official template switch, never with an injected instruction such as `/no_think`.

## 3. Suite

Every item of every benchmark is used, with no subsampling: 33,485 items in total. Each source is pinned to a revision and file checksum.

| Category | Benchmark | Source and split | Items | Answer type | Metric | Licence |
|---|---|---|---:|---|---|---|
| Knowledge | MMLU-CF | `microsoft/MMLU-CF` validation, 14 subjects | 10,000 | letter (A–D) | accuracy | CDLA-Permissive-2.0 |
| Knowledge | ARC-Easy | `allenai/ai2_arc` test | 2,376 | letter | accuracy | CC BY-SA 4.0 |
| Language Understanding | HellaSwag-Pro | official repository, English, all 7 variants × 800 | 5,600 | letter (4–5 options) | accuracy | none stated |
| Language Understanding | Global PIQA | English, non-parallel (100, 2 options) + parallel (103, 4 options) | 203 | letter | accuracy | CC BY-SA 4.0 |
| Math & Reasoning | GSM8K-Platinum | test | 1,209 | number | accuracy | MIT |
| Math & Reasoning | MATH-500 | test | 500 | math expression | accuracy (equivalence) | MIT |
| Tools & Automation | When2Call | multiple-choice test set | 3,646 | letter (A–D) | accuracy | CC BY 4.0 |
| Tools & Automation | ToolAlpaca | real + simulated evaluation sets | 170 | tool calls (JSON) | exact match per case | Apache-2.0 |
| Coding | MBPP+ | EvalPlus release v0.2.0 | 377 | Python code | pass@1 | Apache-2.0 / CC BY 4.0 |
| Coding | EvoEval | release v0.1.0, all 7 subsets | 824 | Python code | pass@1 | Apache-2.0 |
| Classification & Extraction | Banking77 | test | 3,080 | label (77) | accuracy | CC BY 4.0 |
| Classification & Extraction | CLINC150 | test + out-of-scope test | 5,500 | label (150 + `oos`) | accuracy | CC BY 3.0 |

- **Whole benchmarks only.** Every variant and subset is included, so the choice of which parts to use cannot tilt the result.
- **When2Call is used in its multiple-choice form.** The generative form needs an LLM judge, which [Principle 3](#1-principles) rules out.
- **HellaSwag-Pro's repository has no licence file.** It is used for evaluation only; its items are not redistributed.

**Option order.** Published option orders are not always balanced (in HellaSwag-Pro Sentence Sorting the answer is the first option in 798 of 800 items), which would reward position bias. So every multiple-choice item's options are shuffled with a fixed seed derived from the item, identical for every entry. Options that refer to other options ("none of the above") keep their position, and items whose options refer to others by letter ("both A and B") are not shuffled.

**ToolAlpaca.** The model sees the API's documentation and the instruction, and replies with the calls it would make. A case is correct if the predicted calls match the gold calls one to one, in any order: same function names, every required gold parameter present and equal, and every optional parameter present in both calls equal. Credentials (API keys, app IDs) are never scored.

**Exclusions.** 55 items are excluded under objective rules, the same for every entry:

| Rule | Items |
|---|---:|
| ToolAlpaca: a gold call depends on the result of an earlier call | 25 |
| ToolAlpaca: a gold call uses a function the API does not document | 15 |
| ToolAlpaca: gold arguments are not valid JSON | 4 |
| The reference solution fails the official evaluator (EvoEval HumanEval/32 and /130 in Verbose and Concise; MBPP+ 255) | 5 |
| The prompt is longer than 4,096 tokens on an eligible tokenizer (When2Call) | 6 |

## 4. Prompting

- **Zero-shot everywhere.** Few-shot examples interact differently with thinking and non-thinking modes, and they use up short context windows.
- **One user message, no custom system prompt**, rendered through the model's official chat template. Any default system text the template inserts itself is kept.
- **One instruction per answer type**, shared by every benchmark of that type:

| Answer type | Required final form |
|---|---|
| letter | Question, then options as `A. …` lines, then: *"Answer with the letter of the correct option. End your reply with a line of the form "Answer: X", where X is the letter."* |
| number | *"Solve the problem. Put your final answer, a number only, in `\boxed{}`."* |
| math expression | *"Solve the problem. Put your final answer in `\boxed{}`."* |
| label | One instruction line, the text, then all labels one per line in alphabetical order, then: *"End your reply with a line of the form "Answer: <label>", using one label from the list exactly as written."* CLINC150 adds *"If none of the other labels applies, answer oos."* |
| code | The task in a ```python block, then: *"Write the complete Python function `<name>` in a single ```python code block."* |
| tool calls | The API's functions, the user request, then: *"Reply with the API calls needed to fulfil the request, as a JSON list of objects of the form {"name": …, "arguments": {…}}, in the order they should be made. Do not answer the request yourself."* |

HellaSwag-Pro and Global PIQA items start with one fixed stem: *"Choose the option that best completes or answers the following."* Tool tasks use this shared text format rather than each model's native tool-calling template, so every model gets identical inputs and one parser.

## 5. Decoding and inference

| Setting | Value |
|---|---|
| Decoding | The model's own recommended settings for the entry's mode (see below) |
| Samples per item | 4 with fixed seeds; 1 when the settings are greedy |
| Generation budget | `min(32,768, context window − prompt length)` new tokens, for every entry and benchmark |
| Stop | End-of-sequence or the template's end-of-turn token only |
| Precision | bfloat16 |
| Engine | vLLM 0.31.0 |
| Prompt tokens | The rendered template is tokenized without adding special tokens. If the tokenizer expects a BOS token and the template omits it, it is added once (this affects MiniCPM4-0.5B only) |

**Why 32,768 tokens.** It is the output length Qwen3 recommends for reasoning tasks and far above what non-reasoning models use, so one budget limits nobody by design. Truncated answers score 0 and are reported per entry and benchmark. The budget is never changed for one entry alone.

**Decoding settings come from the model maker.** Greedy decoding is not neutral: some reasoning models' cards (e.g. Qwen3) advise against it because it causes repetition loops. Each entry therefore uses its vendor's settings from the pinned revision, in this order:

1. **Mode-specific values stated as recommended in the model card.** Only fixed numeric values count, and values that only appear in a code example do not. The quoted text and link are recorded with the entry.
2. **Otherwise, the model's `generation_config.json`.** If it enables sampling, its temperature, top-p, top-k, min-p and repetition penalty are used, with library defaults for anything it omits.
3. **Otherwise, greedy.**

Settings are never tuned or overridden. A sampled entry's score per item is the mean over its 4 seeded samples. Loops run to the budget and score 0, with no early stopping.

## 6. Answer extraction

Extraction is identical for every entry and depends only on the answer type.

1. **Remove reasoning.** Only the text after the last closing reasoning delimiter (e.g. `</think>`) is parsed; trailing end-of-turn markers are removed. A reasoning block that never closes counts as truncated.
2. **Extract by answer type.** The first rule that matches wins.
   - **letter.** The last `Answer: X` or `answer is X`; otherwise a single letter in the last `\boxed{}`; a reply that is just one letter; a letter followed by exactly that option's text; or a reply that is exactly one option's text. Two letters given together (`B or C`) are wrong.
   - **label.** The rest of the line after the last `Answer:`, or a reply that is just one label. Case, `_`, `-` and outer punctuation are normalised; the result must equal exactly one label. No fuzzy matching.
   - **number.** The last `\boxed{}`, else the last `Answer:` line, else the last number; compared numerically.
   - **math expression.** The last `\boxed{}`, else the last `Answer:` line; judged equivalent to the gold answer by `math-verify`.
   - **code.** The first fenced block that defines the required function; else the longest fenced block; else the whole reply.
   - **tool calls.** The first JSON list of call objects, or a single call object, anywhere in the reply. Argument values are compared after normalisation (trimmed, case-insensitive strings; numeric numbers; lists as multisets).
3. **Every failure is logged with a reason:** unparseable, truncated, unsupported or error.

## 7. Code execution

- Pass@1, averaged over the entry's samples.
- Each program is the task's own preamble (imports, helpers) followed by the extracted code, run with the official evaluators: EvalPlus 0.3.1 for MBPP+, and EvoEval 0.1.0 with EvalPlus 0.2.0 for EvoEval, at their default time limits.
- A sample passes only if every test the official evaluator runs passes.
- Code runs in an isolated sandbox with no network access.
- The reference solutions pass every non-excluded task (1,201 of 1,201).

## 8. Scoring

**Chance level for each benchmark** is the better of two no-skill strategies, computed from the gold labels:

- **The best constant answer**, always giving the same letter or label. This makes a model that always gives the same answer score 0 even when labels are unevenly spread. On CLINC150 it is always answering `oos`: 1,000 / 5,500 = 0.1818.
- **Uniform random guessing** among each item's options, which matters when option counts vary within a benchmark.
- Open-ended answer types (number, math expression, code, tool calls) have a chance level of 0.

| Benchmark | Chance | Rule |
|---|---:|---|
| MMLU-CF | 0.2563 | best constant answer |
| ARC-Easy | 0.2618 | best constant answer |
| HellaSwag-Pro | 0.2429 | uniform guessing |
| Global PIQA | 0.4187 | best constant answer |
| When2Call | 0.2680 | best constant answer |
| Banking77 | 0.0130 | best constant answer |
| CLINC150 | 0.1818 | best constant answer (`oos`) |
| GSM8K-Platinum, MATH-500, ToolAlpaca, MBPP+, EvoEval | 0 | open-ended |

**Formulas:**

```
item_i        = mean over the entry's samples of correct (1) / wrong (0)
raw_b         = mean of item_i over all items in benchmark b
skill_b       = clip((raw_b − chance_b) / (1 − chance_b), 0, 1)
category_c    = mean of skill_b over the benchmarks in c
Nano LM Index = 100 × mean of the 6 category scores
Raw index     = the same with raw_b in place of skill_b
```

**Confidence intervals and ranks.**

- 95% bootstrap confidence intervals come from 10,000 resamples of items within each benchmark, carried through to the categories and the index. Every entry is resampled with the same items, so comparisons are paired.
- Entries are ranked by their Nano LM Index, highest first; only exactly equal scores share a rank. The interval is shown next to every score, so readers can see when two entries are close.

## 9. What is published for every entry

- The Nano LM Index with its confidence interval, the raw index and the six category scores.
- For each benchmark: skill, raw accuracy, chance level and item count.
- Mean output length and the share of answers that hit the token budget.
- Model repository and pinned revision, mode, parameter count, context window, licence and engine.
- Decoding settings and where they came from (the model card quote and link, `generation_config.json`, or greedy), and the number of samples.

## 10. Editions

An edition freezes the suite, prompts, extractors, budget and chance levels. Index numbers are comparable only within one edition.

| Change | Edition | Action |
|---|---|---|
| Extractor or scoring bug fix | patch: V1 → V1.0.1 | Every stored output is rescored; nothing is regenerated |
| Add, remove or replace a benchmark; change a prompt or the budget | new version: V1 → V2 | Every entry is run on the new edition before the board switches |
| New model | none | Run on the current edition |
| New model revision | none | A new entry; the old one is kept and marked superseded |

**Admitting a new benchmark.** It must be scorable deterministically, licensed for evaluation, pinnable to a revision, fit in 4,096 prompt tokens on every eligible tokenizer, and belong in exactly one category. Benchmarks weigh equally within a category, and categories weigh equally in the index.

**Checks before a board is published.**

- Unit tests for every extractor, including adversarial answer formats.
- Synthetic entries confirm the scoring: a gold oracle scores 100.00, the best constant answer 0.00 and uniform random 0.14.
- Failures are sampled per benchmark across entries and reviewed by hand. A failure caused by the extractor rather than the model is fixed in a patch edition, and every entry is rescored.

## 11. V1 entries

| Entry | Parameters | Context | Decoding (source) |
|---|---:|---:|---|
| Qwen3-0.6B `think` | 596,049,920 | 40,960 | T 0.6, top-p 0.95, top-k 20, min-p 0 (model card) |
| Qwen3-0.6B `no-think` | 596,049,920 | 40,960 | T 0.7, top-p 0.8, top-k 20, min-p 0 (model card) |
| Falcon-H1-0.5B-Instruct | 521,411,104 | 16,384 | greedy |
| MiniCPM4-0.5B | 433,873,920 | 32,768 | T 0.8, top-p 0.8, top-k 50 (generation_config) |
| Qwen2.5-0.5B-Instruct | 494,032,768 | 32,768 | T 0.7, top-p 0.8, top-k 20, repetition penalty 1.1 (generation_config) |
| Granite-4.0-H-350M | 340,332,224 | 32,768 | greedy |
| Granite-4.0-350M | 352,379,904 | 32,768 | greedy |
| ERNIE-4.5-0.3B-PT | 360,748,032 | 131,072 | T 0.8, top-p 0.8, top-k 50 (generation_config) |
| LFM2.5-350M | 354,483,968 | 128,000 | T 0.1, top-k 50, repetition penalty 1.05 (model card) |
| LFM2-350M | 354,483,968 | 128,000 | T 0.3, min-p 0.15, repetition penalty 1.05 (model card) |
| Baguettotron `think` | 320,956,992 | 4,096 | greedy |
| SmolLM2-360M-Instruct | 361,821,120 | 8,192 | greedy (the card's values appear only in a code example) |
| Gemma 3 270M (it) | 268,098,176 | 32,768 | T 1.0, top-p 0.95, top-k 64 (generation_config) |

**Short context windows.** Baguettotron (4,096 tokens) and SmolLM2-360M (8,192 tokens) have little room for long answers, so more of their answers reach the limit and count as truncated. These are limits of the models themselves and are reported, not adjusted for.