Combining ML, NLP and LLMs

Toolkit · Quantitative

A measurement-design toolkit with a searchable methods catalogue, structured specifications, and a database interface.

What has been built

A methods catalogue, measurement specifications and a validator support research design. A teaching card documents musical typicality. The proposed product-description measure has no scores.

What comes next

Develop a measurement workflow and agent skill; evaluate functional differentiation using independent judgments, paraphrases and alternative comparison sets.

New wording or a new function?

The design case is novelty in product descriptions, defined as described functional differentiation: how far the functions and relationships a description states differ from those stated in a bounded set of earlier descriptions.

A rare word or a distant vector is evidence about a representation; whether a function differs has to be shown separately.

Three fictional descriptions

Earlier description
Schedules appointments from a shared calendar.
The reference.
Reworded
Books meetings using a team calendar.
Different words; the stated function is the same.
Focal description
Schedules appointments and routes exceptions to a supervisor for approval.
Adds a stated function: an approval relationship.

Written for the design: no model labelled these sentences and no score is computed. The added relationship is a stated difference only; it does not show that the product works, or that no eligible earlier description states it.

Explore this application in the catalogueProducts and functional concepts

See the proposed workflow and validation checks

The proposed sequence

This six-step workflow and its agent skill remain proposed. Extraction, representation and comparison would be checked separately.

Keep the record
Draws on acquired texts
Each description keeps its source, entity, retrieval date and version. A posting date does not show that the retrieved wording existed on that date.
Extract the stated functions
Draws on LLMs
LLM annotation records the actors, tasks, functions and conditions a description states, each with its supporting passage; what is not stated stays unknown. The result is a set of observations, not a verdict on novelty.
The LLM-annotation workflow gives the pattern for such records; the two have not been connected in a run.
Choose a representation
Draws on NLP
The same texts can be represented as weighted words, topics, embeddings, or extracted functions and relationships. Each keeps different information.
In the catalogueLexical first use dictionaries and topics Semantic geometric distances Concept claim event and narrative structure Multimodal artifact comparison
Fix the comparison set
Set by the researcher
The researcher decides which earlier descriptions are eligible: the time window, the exclusions, and how versions, repeats and self-matches are treated. A missing predecessor can make a description look new.
In the catalogueReference universes and time
Compare, keeping readings apart
Draws on statistics, ML and LLMs
Distance to the nearest earlier description, average distance, rarity, unusual combinations, a model’s surprise and an LLM’s judgment answer different questions; none of them alone shows a different function. A structured comparison keeps the matched functions, the remaining differences and the uncertainty with any number.
Computational baselinesDensity rarity and outlier measures Recombination and relational null models Surprise likelihood and information Training-corpus overlap memorization and authenticity
LLM assessmentDirect LLM rubric scoring Pairwise and ranking judgments Retrieval grounded prior comparison Decomposed contribution verification Supervised and calibrated evaluators Ensembles and human assisted assessment
Test the reading
Draws on human judgments
Extraction, retrieval and comparison are each checked, against independent human judgments, deliberate changes to the texts and separate calibration and test samples, before a score supports a substantive claim.
In the catalogueProspective validation and empirical follow up Validation uncertainty and inference

The checks it calls for

Specified for later empirical work; none has been run.

Change to the texts	What it asks	Result
Reword, keeping the meaning	Does the measure ignore a change of wording alone?	Not run
Change a function or relationship	Does it detect the difference it claims to measure?	Not run
Add or omit a relevant predecessor	Does the apparent novelty depend on a missing reference?	Not run
Change style, length or status cues	Does presentation leak into the measure?	Not run
Change the window, category or corpus size	How far does the result depend on the comparison set?	Not run
Change the model or prompt	Do the results hold across versions?	Not run

Read the AI prompt

# Explore a text-measurement design

Help me explore TextData’s **Combining ML, NLP and LLMs** prototype.

Read the [public page](https://textdata.info/toolkit/combining-ml-nlp-llms/) and the [measurement appendix](https://github.com/joyzhzh/textdata/tree/main/appendix/09-measurement). In that folder, start with `README.md`, `data/catalogue/records.jsonl`, `example/input/ASKIN_MEASURE_CARD.jsonl` and `docs/g9-design/EXTRACTION_AND_VALIDATION.md`; consult `tool/schemas/measure-card.schema.json` and `tool/scripts/validate_measure_cards.py` when explaining the card. Cite the files you actually read and say if one is unavailable.

1. Explain the page’s fictional calendar example: why rewording appointment scheduling differs from adding supervisor approval, and why neither proves market novelty. Do not invent scores.
2. Help me use the catalogue to explore **Products and functional concepts (APP05)** and one relevant method, such as Decomposed contribution verification (MF11), which includes functional-equivalence checks. Explain the comparison set and a concrete failure case. The method–application pairs are unassessed.
3. Walk through the musical-typicality teaching card: the earlier-song reference window, reported values and explicit unknowns. If a local checkout and Python are available, show or run the README’s two validator commands and explain why structure passes while submission is refused.
4. Apply the example to my question with a short draft measurement specification: construct, observed text, comparison population and time window, candidate representation, and one validation check. Label new choices as proposed. The catalogue and validator work now; the product-description scoring workflow and agent skill remain proposed. No text scores or validated estimates are supplied, and structural validation does not establish scientific validity.

My research question or text-measurement task: [fill in here]

Explore the catalogue

Explore methods, application settings and research questions before choosing a text measure. Search “functional equivalence” or open the product-description setting from the illustration.

The 110 records describe design options. All 490 application–method pairs remain unassessed and unsearched; the catalogue recommends no method.

Methods catalogueWorking design prototype

Search the recordsRecord typeResearch area

14 of 110 records

Lexical first use dictionaries and topics

Method family MF01 · Designed; applicability unassessed

Question

What do lexical and embedding-distance baselines estimate, and how sensitive are they to representation and comparator choice?

Subareas

dictionaries
first use
TF IDF
topics
LSA
document encoders
mean nearest and centroid distances
pretraining-scale n-gram overlap (cross-ref L56)

Adverse case

Synonyms, polysemy, anisotropy, hubness, normalization and converting similarity into novelty without a construct argument.

Planned output

Formula and parameter atlas separating mean, nearest, kth-neighbor and centroid measures. Contribute mechanism ancestry and dated derivation links to the shared measure genealogy; L50 owns the integrated genealogy.

Related records

Question, subareas, adverse case and planned output describe the owner lane; some lanes serve more than one method family.

Source: docs/g9-design/METHOD_FAMILIES.json, method_id=MF01 · Detail from: docs/g9-design/LANES.jsonl, lane_id=G9-L34

Files

Complete catalogueJSON, 375 KB: all tables, with the 490 pairs and the six registry specifications
Browse recordsJSONL, 126 KB: the 110 records, one per line
Application–method pairsCSV, 25 KB: the 490 pairs, one per row
DatabaseSQLite, 385 KB: the same tables in one file

The downloadable data match the measurement appendix.

The measure card

A measure card specifies the construct, inputs, comparison set, scoring and validation. Fields distinguish reported values, explicit unknowns and proposed choices. The validator checks structure and refuses a teaching card as a submission.

The teaching card

The one card filled in documents the measure of musical typicality in Askin and Mauskapf (2017): each song is compared with the songs that charted in the 52 weeks before its debut, weighted by genre similarity. It is unreviewed teaching material. Typicality is measured from sonic features, so the card shows the format, not a text measure.

Inspect the card fields

44 fields in nine sections: 17 reported, 24 unknown, 3 proposed.

Identity
5 fields
- Citation: Reported
- Manifestation: Reported
- Publication status: Unknown
- Corrections retractions: Unknown
- Access and inspection: Reported
Construct
5 fields
- Intended construct: Reported
- Level and unit: Reported
- Text role: Reported
- Domain and language: Unknown
- Relative to whom: Reported
Inputs
5 fields
- Text or features: Reported
- Preprocessing: Reported
- Representation: Reported
- Model version training cutoff: Unknown
- Prompt and decoding: Unknown
Scoring
8 fields
- Formula or pseudocode: Reported
- Comparator universe: Reported
- Time window: Reported
- Restriction and weighting: Reported
- Null model and support: Unknown
- Aggregation: Reported
- Direction and normalization: Reported
- Missingness: Unknown
Study design
5 fields
- Population and sample: Unknown
- Unit denominator: Unknown
- Raters rubric blinding: Unknown
- Clustering and dependence: Unknown
- Splits and leakage: Unknown
Validation
4 fields
- Target: Unknown
- Design and population: Unknown
- Estimates and uncertainty: Unknown
- Transport and invariance: Unknown
Rival explanations
3 fields
- Confounds: Unknown
- Adverse evidence: Unknown
- Author limitations: Unknown
Reproducibility
5 fields
- Code: Unknown
- Data: Unknown
- Model: Unknown
- License and rights: Unknown
- Execution status: Proposed
Interpretation
4 fields
- Supported scope: Reported
- Our inference: Proposed
- Unresolved details: Unknown
- Reuse and correction lineage: Proposed

Unknown fields stay empty until their source support is established.

Read the scoring fields and validator example

The scoring section as recorded

Field	Status	Recorded value	Where in the source
Formula or pseudocode	Reported	Compute song-pair cosine similarities; multiply by genre similarity; average weighted entries.	Author manuscript, printed pp. 18–20 (PDF pp. 18–20); Typicality subsection.
Comparator universe	Reported	Charting songs in the preceding 52 weeks before focal debut.	Author manuscript, printed pp. 18–20 (PDF pp. 18–20); Typicality subsection.
Time window	Reported	52 weeks.	Author manuscript, printed pp. 18–20 (PDF pp. 18–20); Typicality subsection.
Restriction and weighting	Reported	Genre similarity weights; same-genre weight one.	Author manuscript, printed pp. 18–20 (PDF pp. 18–20); Typicality subsection.
Null model and support	Unknown	Left empty. Not inspected for this teaching example.
Aggregation	Reported	Average weighted pair similarities.	Author manuscript, printed pp. 18–20 (PDF pp. 18–20); Typicality subsection.
Direction and normalization	Reported	Greater similarity means greater typicality.	Author manuscript, printed pp. 18–20 (PDF pp. 18–20); Typicality subsection.
Missingness	Unknown	Left empty. Not inspected for this teaching example.

What the validator does

Try the two Python commands in the appendix README.

Input	Result
The teaching card, checked for structure	Accepted
The teaching card, offered as a submission	Refused: teaching material cannot be submitted
A copy with a value in a field marked unknown	Refused

The validator checks structure only. It cannot tell whether the cited page supports the value recorded.

Use and limits

Limits

The product-description specification is unpopulated; no text has been scored or estimate validated.
The musical-typicality card is unreviewed teaching material about a non-text measure. Structural checks do not establish source support or construct validity.
The measurement workflow and agent skill remain proposed, including evaluation with independent judgments, paraphrases and alternative comparison sets. A reference-text cutoff does not remove later knowledge from a model.

In the repository

Run the validator, open the offline catalogue, inspect the design tables, or adapt the measure-card schema. The README gives commands and expected results.

Open measurement appendix

Sources and credit

Measure documented by the teaching card: Askin and Mauskapf (2017), “What Makes Popular Culture Popular? Product Features and Optimal Differentiation in Music”, American Sociological Review 82(5): 910–944. The card holds the project’s own notes on the measure, with page locators; it reproduces no text from the paper.

The catalogue data and the project’s teaching-card notes are licensed under CC BY 4.0. The cited third-party works retain their own terms.

Collection provider: Product Hunt.

Method sources: Grimmer and Stewart (2013), on validating automated text methods for each research problem; Reimers and Gurevych (2019), on sentence embeddings for similarity, whose results do not validate this construct.