Large-scale data acquisition
Toolkit · Quantitative
A reusable AI workflow for assembling, cleaning, and verifying large textual collections.
What has been built
An inventory covers 806,941 historical document records. A rebuilt workflow matched 191 identifiers and 169 raw files in a two-day comparison. Two new official-site retrievals yielded identical reading copies for all 191 documents.
What comes next
Integrate acquisition skills and test additional sources. Investigate the 22 raw-file discrepancies; full historical replay remains unavailable.
From public text to a sampling frame
Define a publication window, retrieve its documents, retain gaps and prepare verified reading copies. For a study of agency obligations, the inventory distinguishes missing text from a document with no relevant obligation.
Open acquisition appendix for the working adapter, inventories and offline example.
Explore the working workflow
Choose a stage to see what it produces. The Federal Register adapter uses fixed rules; it does not classify or interpret the documents.
Define the source
Set a publication window and document identity before retrieval.
Example: 7–8 December 2020, with no agency or keyword filter.
Retrieve
List the provider’s documents, check each response and record missing text.
Example: 191 listed documents with text available from both official sites.
Inventory
Keep one row per document, including gaps, source and byte fingerprint.
Example: The same 191 identifiers as the historical inventory.
Prepare reading copies
Remove HTML while retaining document headers. Cleaning does not restore missing images.
Example: 191 reading copies; supplied text is email-redacted.
Package
Group text into monthly packs with a corresponding inventory.
Example: A small pack ready for offline reuse.
Verify and compare
Check bytes against inventory fingerprints and compare document identities.
Example: The original comparison found 169 raw matches and 22 unresolved differences.
Run the supplied example offline, or use the workflow instructions to define a new retrieval window.
A two-day example
For 7–8 December 2020, the workflow recovered all 191 identifiers in the historical inventory. Of the original raw files, 169 matched and 22 differed for an unresolved reason. The two official sites yielded identical reading copies for all 191 documents.
Try the appendix’s offline example to follow a document from inventory to reading copy and verified pack. The supplied copies remove email addresses; their replay gives 61 matches and 130 differences against historical fingerprints, separate from the original 169/22 finding.
Explore the collection
The inventories cover 806,941 Federal Register document records from January 2000 to 15 September 2026. A record is a publication date plus document number, not a regulatory event.
All 321 monthly inventories are supplied in the appendix. Historical text packs are not distributed; the two-day example provides a working text sample.
| Document records | 806,941 |
|---|---|
| With text | 806,237 |
| Without text | 704 |
| Publication dates | 3 January 2000 to 15 September 2026 |
Missing text stays in the sampling frame.
Inventories by year
Each bar is one year of monthly inventories; 2026 runs to 15 September. Choose what the bars count, then select a year to open its months and a month to open its record, as the folder’s inventory index and pack list give them.
Highest: 2000, 33,325 document records
Select a year to open its monthly inventories.
Download the 321 monthly records:CSV, 67 KBJSON, 121 KB
Each row joins a month’s line in the folder’s inventory index with its line in the pack list; like those files, the rows are under CC BY 4.0. The inventories themselves, one line per document, and the texts are not served by this site.
Scope and next steps
The workflow is demonstrated on one provider and two publication days. Historical text packs are not supplied, and the original listing/download programs were not retained, so full historical acquisition cannot be replayed. Images remain placeholders; file integrity does not establish content completeness or meaning.
Integration with research skills, additional-source adapters and investigation of the 22 raw-file differences remain proposed.
Sources: the Office of the Federal Register and the U.S. Government Publishing Office, through FederalRegister.gov and GovInfo. Federal Register material may be republished under 1 CFR 2.6; embedded third-party material retains its rights. Project code is Apache-2.0 and project-authored data/documents are CC BY 4.0. The appendix notice gives attribution and release modifications.