# How the collected papers annotate text and test multiple models

**Methods synthesis, 20 October 2026.** The useful common pattern is a defined coding task, fixed text inputs, recorded model outputs and an explicit comparison procedure. However, **comparing models, repeating a model and combining models are different operations**. A paper that benchmarks many models has not necessarily tested a voting ensemble.

This assessment examines relevant methods in seven papers and three supplements and extends the [workflow assessment](workflow-literature.md). It reconstructs reported methods; it is not a replication or systematic review. Source versions, page references and uninspected materials are qualified below.

## The methods at a glance

| Study | What the LLM receives and returns | How multiple models or repetitions are used | What evaluates the output? |
|---|---|---|---|
| **Gilardi, Alizadeh & Kubli (2023)** | One text plus the human coding instructions; returns a task-specific relevance, stance, topic or frame label. Fresh chat for each tweet. | **One model**, `gpt-3.5-turbo`, twice at temperature 1 and twice at 0.2. Measures repeat agreement; no majority ensemble. | Trained-human labels on cases where two annotators agree; also compares crowd workers. |
| **Carlson & Burbano (2026)** | One Kickstarter description plus a prompt; returns one binary social-benefit label. | **5 models × 7 prompts**, evaluated separately on the same 250 reference cases; later 100 prompt variants. | Human-consensus accuracy and error proportions, then sensitivity of a downstream regression to alternative labels. |
| **Çelebi & Penczynski (2026)** | Experimental messages plus codebook components and optional examples; returns schema-constrained JSON. | **4 models**, crossed with prompt content, format, framing, examples and reasoning settings. Separate-model benchmarks, not a combined vote. | Same-item agreement with agreed human labels, paired comparisons and uncertainty intervals. |
| **Soria, retained v2-associated manuscript** | One survey response, its question and six category definitions; returns six binary JSON labels. | **16 models** first benchmarked individually; saved predictions then combined using majority or unanimity, including all 560 three-model combinations. | Two trained coders' adjudicated reference; precision, recall, accuracy and response-level bootstrap. |
| **Cheng, Mayya & Sedoc, SILICON v4** | Actual documents/utterances plus developed coding guidelines; returns task labels, including multi-label sets. | **7 models** benchmarked; task-specific top-three ensembles. Five repeated focal-model outputs estimate concentration for routing uncertain cases to auxiliaries. | Human-reference agreement; compares individual models and routed ensembles. Model/threshold selection limits the interpretation of the best observed result. |
| **Camuffo et al., preliminary v3** | Fixed venture descriptions plus a judgment/rating question; returns A/B or numeric scores. | Repeated calls under changed wording, model, format, order or scoring procedure. Main demonstrations diagnose variation; the broader ensemble workflow is recommended separately. | Output distributions and agreement; no adjudicated human accuracy benchmark for the core fixed-venture demonstrations. |
| **Messing v6** | Depending on the demonstration, text/response context for a judge, or a question for a model to answer. | Crosses model, prompt, temperature and repetitions. Arena explicitly averages **3 judges × 5 prompt variants**; other demonstrations do not establish a per-item majority vote. | Pipeline variance; human preferences/reference labels or an answer key where appropriate. Reference types and denominators differ across tasks. |

## 1. Human codebooks become annotation instructions

**Gilardi et al.** supply the same task instructions to the LLM, trained coders and crowd workers. The retained supplement gives category meanings and explicit exclusions: for example, discussing content moderation differs from merely being content that might be moderated. The model reads each tweet separately and receives a fresh chat, avoiding earlier annotations in its context. Across four datasets there are 6,183 texts, but tasks differ by dataset. Two responses at each of two temperatures yield a consistency comparison; these four responses are not described as a final four-vote label. The paper does not establish a strict JSON/retry protocol in the methods inspected. Accuracy excludes human-disagreement cases, so it is not accuracy over every ambiguous text. **Locators:** main PDF pp. 3–6; included codebooks PDF pp. 8–13.

**Çelebi and Penczynski** make the prompt construction more explicit. They remove administrative content from human codebooks, separate task context, theory, category rules and examples, then assemble controlled variants. Their template contains the task, optional persona, context/theory, category definitions, label codes, examples, constraints and JSON output instruction. Claude Opus 4.6 helps generate variants; the evaluated classifiers are GPT-5.4, GPT-5.4-nano, Qwen-3.5-397B and Qwen-3.5-9B. Prompt-authoring and classification models have different roles.

Their three reported tasks contain 1,494 player utterances from 719 chats, 493 voting-game messages and 851 coordination-game messages. A chat packet can contain multiple players; an evaluated utterance is not necessarily one independent API call. Same task items are compared across conditions. The main template requests labels without explanations. The paper varies codebook detail separately from examples, format/persona and reasoning. It reports 703,253 calls, 702,809 successful, across 1,182 combinations; we have not reproduced those totals. **Locators:** main pp. 4–8, Figure 1 p. 5; results pp. 9–12; S8 pp. 1–2; S10 pp. 1–4.

Two implementation qualifications matter. S8 sets temperature 0 for non-reasoning and 1 for reasoning, so off-versus-on changes both settings. It describes per-item JSONL records, error flags, retries and endpoint/provider rotation; parallel request batches do not mean multiple texts share one context. The main paper's same-item statistical comparison does not by itself estimate repeated-run variance. Its saved S6/S8 also contain task/model-scope discrepancies and unresolved references, so an exact executed factorial matrix cannot safely be reconstructed from their prose alone.

**Transfer:** a usable codebook specifies what qualifies, what does not, and difficult boundary cases. More reasoning cannot be assumed to compensate for a missing definition. Meaning-preserving wording changes, added examples and a genuinely broader relevance rule are distinct interventions. New examples or rules create a new instrument version.

## 2. A multi-model benchmark need not combine the models

**Carlson and Burbano** automate four steps: load a campaign description, prepend the chosen task prompt, call an API and append the final 0/1 label. Their five models are GPT-4, GPT-4o, GPT-4o-mini, Claude 3 Opus and Claude 3.5 Haiku, at temperature 0. Seven prompting strategies produce 35 model-prompt conditions on the same 250 expert-consensus cases. The full source database has 295,985 projects, but running the chosen classifier on that entire database is a proposed deployment, not evidence that all 35 conditions were run corpus-wide. **Locators:** PDF pp. 10–16 / printed pp. 708–714; Table 3 PDF p. 13.

Their named self-consistency prompt asks a **single model within one call** to consider three perspectives and return a final answer. It does not establish three fresh sessions or three separately retained votes. Similarly, the tree-of-thought prompts are written instructions; the inspected method does not demonstrate an external tree-search implementation. This differs from an ensemble with three independently saved reader outputs.

They subsequently generate 100 prompt variants and inspect how alternative annotations change a funding-success regression. They warn against selecting a prompt for a preferred downstream finding. The 250 cases are held out from an earlier supervised model's training, but also inform the LLM comparison and choice; a second untouched test of the selected LLM configuration is not established in the inspected article. No explicit repeated-call factor or complete failure/retry policy is supplied there. **Locators:** PDF pp. 16–22 / printed pp. 714–720.

**Transfer:** compare each reader as well as the majority, preserve alternative outputs, and test how alternative annotations affect downstream estimates. Without human reference labels, such comparisons remain sensitivity/agreement analyses. Choosing the highest agreement, largest list or most favorable regression does not establish annotation accuracy.

## 3. Actual ensembles combine saved outputs under a stated rule

**Soria** is a direct example. Each of 16 models reads the same survey response, question and detailed codebook, independently returning a six-component binary vector. It uses detailed definitions but reports no labeled few-shot examples or chain-of-thought instruction. Most models use temperature 0; GPT-5 and GPT-5 Mini use temperature 1 with minimal effort. These are disclosed provider-dependent settings, not perfect equality of every decoding control. **Locators:** pp. 6–13, 52–54.

The study first evaluates individual models, then combines the stored labels. Majority requires more than half positive; unanimity requires every member positive, otherwise negative. The 560 triples formed from 16 models reuse predictions and overlap extensively. They are **560 computed ensembles, not 560 fresh replicated experiments**. Separate temperature and effort checks make additional calls; the minimal-versus-high effort comparison reuses the original minimal predictions. **Locators:** pp. 12–13, 65–68, 90–92, 98–99.

For its main common analysis set, the paper excludes 28 responses with incomplete predictions across models, leaving 3,180 responses / 19,080 category cells. It compares against two coders' adjudicated labels and resamples whole responses, keeping their six category decisions together. That common-denominator design is useful, but applications must also retain explicit failures and explain any excluded inputs. The exact full prompt templates are deferred to Supplement S1, which was not found in the collected materials. We can describe their components, not claim byte-for-byte reproduction.

**SILICON** tests another ensemble design. It first benchmarks seven models across nine annotation tasks, using actual documents, reviews, posts and utterances. Coding guidelines are developed through independent human coding, discussion and checks on fresh samples. The main model comparison holds a chosen task prompt fixed; choosing that prompt using GPT-4o is itself part of the reported selection procedure. **Locators:** main/PDF pp. 13–14, 17–26; appendix PDF pp. 45–49.

For each task it then chooses the strongest model as the focal classifier and the next two as auxiliaries. Five independently sampled focal-model responses per item at temperature 1 produce the frequency difference between the most common and second-most common answers. A threshold determines which uncertain items go to the auxiliaries for majority voting; other items retain the focal label. Routing helps some tasks and hurts others. The five outputs estimate focal-model concentration; they are not five replicated three-model ensembles. The same reference data inform ranking and threshold search, so the best observed routing point is not independently established out-of-sample performance. **Locators:** main pp. 16–17, 29–32; appendix PDF pp. 43–44.

**Transfer:** test whether the vote improves on its members rather than assuming it does. Reference-relative accuracy needs a reference. Routing requires its own decisions and validation. Recombining existing reader outputs is an arithmetic comparison, not a fresh trial.

## 4. Repeated calls and factorial diagnostics test variability

**Camuffo et al.** repeatedly evaluate a fixed pair of venture descriptions, or one description on ten criteria. They manipulate wording, roles, format, order, model or separate-versus-joint scoring. Typical designs report 20 repeated Level-1 evaluations or 30 Level-2 evaluations, with a separate 100-draw sampling exercise. These are repeated measurements of a small fixed stimulus set, not that many independent documents. Outputs are constrained A/B choices or numeric scores; distribution comparisons diagnose instability. The broader prompt × sample × model aggregation procedure is a recommendation distinct from these demonstrations. The low/high reasoning comparison also changes justification instructions, so it does not isolate reasoning effort. **Locators:** main PDF pp. 14–24; implementation PDF pp. 57–60.

**Messing** separates several experiments. Its safety study reports 141 response items × 5 prompts × 3 temperatures × 3 judges × 8 repetitions. Arena instead uses 4,676 existing battles, three judges and five prompts at temperature 1, with one call per cell and swapped response order for pairwise judging. Its actual Likert ensemble averages 15 scores per response; its pairwise ensemble averages 30 order-normalized verdicts with explicit tie thresholds. MMLU is a different design: models answer questions evaluated against an answer key, without an LLM judge layer. **Locators:** main pp. 5–7; disclosure p. 47; Arena pp. 50–51.

Some retained Messing v6 details conflict: the Arena holdout claim and full-sample table denominators, 15-versus-30 scores per response, and general versus task-specific parse rates. Camuffo's cross-model failure accounting is also unclear. These papers illustrate design ideas, but their prose is not an independently verified executable specification. Exact wire-level prompts or raw outputs were not inspected for these two papers.

**Transfer:** separately identify the item, reader/session, model, prompt, setting and repeated ensemble. Labels produced within one session share that session's context. Treating every judgment or every pairwise comparison as an independent experimental replication would overstate information. A change from full-catalogue sessions to one-item calls or fixed chunks changes the execution protocol.

## Implications for annotation design

The reading supports several distinctions, without prescribing a study plan:

- Specify the coding object and codebook before comparing model outputs. Classifying descriptors and classifying source passages are different tasks.
- Preserve individual outputs, any combined vote and explicit failures. A combined label requires evaluation of its own.
- Keep configuration sensitivity, repeated-run variation and error relative to an independent reference separate.
- Separate instrument development from evaluation. Agreement or favorable downstream results alone do not establish annotation accuracy.

## Source limits and navigation

Source PDFs are not included. Exact templates were inspected in Gilardi's included codebooks and Carlson's Table 3; Çelebi's saved supplements expose prompt-generation and API details, with the scope discrepancies described above. Soria's Supplement S1 was not found; SILICON's external exact-task-prompt file and Messing's code-hosted prompts were not inspected. No unobserved prompt, retry policy, missing label or replication count has been invented to fill those gaps.

Additional retained-version caveats: Carlson's consensus reference also informs selection; SILICON's main baseline-size statement differs from task-specific appendix counts, and its strict routing inequality versus full-routing endpoint is ambiguous. Nonsignificant model differences in either study are not proof of equivalence. The [earlier assessment](workflow-literature.md) covers the distinct validity and downstream-inference questions.