AI defects
Where AI can fail in textual inquiry, what TextData checks, and what those checks leave unresolved.
A source link, a passing check and a persuasive interpretation establish different things. Explore the failures separately, then follow each response to the work that supports it.
Words can match.
Meaning can still be wrong.
In the annotation prototype, a substituted actor and an omitted assignment both pass the source checks. The tests expose what quotation matching cannot establish.
See the example and its limitsExplore the failure modes
Choose a mode to see its consequences, the relevant TextData response and the limits of the evidence.
17 failure modes
HallucinationInventing facts or misrepresenting evidence.
Why it matters
An invented reference or a summary that drops a source’s qualification can enter a literature review, codebook or measure.
Why this can happenTextData’s response
Link claims to sources and separate production from review.
The methods-review correction shows why a source must be read again. The separate discovery of fabricated references in citing papers does not establish that AI wrote those references.
Inspect the evidence and limitsReasoning errorsReaching the wrong conclusion from the available information.
Why it matters
A source can be quoted correctly while the analysis assigns responsibility to the wrong actor or treats missing evidence as a negative case.
Why this can happenTextData’s response
Separate quotation checks from judgments of meaning.
The annotation and Gioia examples expose this gap. Neither supplies a population estimate of reasoning errors.
Inspect the evidence and limitsFalse confidenceSounding more certain than the evidence warrants.
Why it matters
A provisional code or an incomplete search can be presented as a settled result.
Why this can happenTextData’s response
Retain unknowns and use researcher-defined stopping criteria.
Unknown fields and qualified stop decisions make uncertainty visible. They do not calibrate a model’s confidence.
Inspect the evidence and limitsIncomplete work and false completionLeaving work undone, then saying it is finished.
Why it matters
A corpus may have missing records even though the agent reports a complete review.
Why this can happenTextData’s response
Retain unknowns and use researcher-defined stopping criteria.
TextData has bounded completeness and structural checks, but these examples do not document a caught false-completion incident.
Inspect the evidence and limitsGaming the goalMeeting a success criterion while defeating its purpose.
Why it matters
A requirement for valid quotations can be satisfied without identifying all the responsibilities in a passage.
Why this can happenTextData’s response
Separate quotation checks from judgments of meaning.
The deliberately incorrect annotation records test the criterion’s weakness. They are not evidence of intentional gaming or a tested defense against it.
Inspect the evidence and limitsOverengineeringAdding work that the task does not need.
Why it matters
Extra agents, layers or repeated checks can consume effort without resolving a remaining research question.
Why this can happenTextData’s response
Bind execution to the approved plan.
Bounded scope and stopping rules are relevant design practices. The maintenance guard is narrower; no TextData example establishes reduced overengineering.
Inspect the evidence and limitsGoal driftSolving a different problem.
Why it matters
A request to compare rival explanations can turn into a persuasive account of just one.
Why this can happenTextData’s response
Bind execution to the approved plan.
The case-method guard detects changes to a sealed maintenance plan’s inputs. It cannot decide whether an evolving analysis still answers the researcher’s question.
Inspect the evidence and limitsForgettingLosing decisions and context.
Why it matters
A resumed analysis can revive a rejected code or lose the reason a category was narrowed.
Why this can happenTextData’s response
Seek contrary evidence and preserve reasons for revision.
Versioned records support reconstruction. TextData has not demonstrated that agents reliably retrieve and apply every prior decision.
Inspect the evidence and limitsInconsistencyChanging the answer without a relevant change in the evidence.
Why it matters
The same passage can receive different codes on different runs, changing a comparison or downstream measure.
Why this can happenTextData’s response
Preserve inputs; evaluate repeated runs and version changes.
Stored requests make comparisons possible. Replayed exports do not establish repeatability of newly generated judgments.
Inspect the evidence and limitsSycophancyAgreeing at the expense of judgment.
Why it matters
An analyst can endorse a researcher’s preferred theory without giving contrary accounts equal attention.
Why this can happenTextData’s response
Seek contrary evidence and preserve reasons for revision.
The qualitative workflows request rivals and contrary evidence. Their same-model, unblinded review is not a demonstrated reduction in sycophancy.
Inspect the evidence and limitsGeneric answersReplacing a specific thought with familiar language.
Why it matters
Broad labels can erase the difference between permission to act and approval after the event.
Why this can happenTextData’s response
Seek contrary evidence and preserve reasons for revision.
The revised interview distinction is a concrete example of added specificity, not a validated general control over generic answers.
Inspect the evidence and limitsBiasJudging people or claims by associations the evidence does not justify.
Why it matters
Names, dialects or familiar organizational categories can influence judgments that should depend on the specified evidence.
Why this can happenTextData’s response
No general safeguards demonstrated.
No TextData demonstration establishes a general safeguard against bias in model judgments.
Inspect the evidence and limitsInformation leakageSharing material beyond its intended audience.
Why it matters
Private interview material can enter a public summary or be sent to an unsuitable external service.
Why this can happenTextData’s response
No general safeguards demonstrated.
Withholding transcripts from one release is a case-specific decision, not a tested general leakage safeguard.
Inspect the evidence and limitsPrompt injectionTaking instructions from the material being read.
Why it matters
A document retrieved as evidence can attempt to redirect an agent’s answer or tool use.
Why this can happenTextData’s response
No general safeguards demonstrated.
TextData has no demonstrated general defense against prompt injection. Source links and quotation checks are not such a defense.
Inspect the evidence and limitsFalse assuranceA review or evaluation that misses what matters.
Why it matters
Passing quotation checks can be mistaken for correct analysis; several approvals can repeat the same omission.
Why this can happenTextData’s response
Separate quotation checks from judgments of meaning.
The reviewer correction and the constructed annotation errors expose both problems. More approving reviews do not alone establish validity.
Inspect the evidence and limitsOpaque explanationsA plausible account that does not establish why the system acted.
Why it matters
An agent’s explanation for excluding a paper may disagree with the record of what it read and did.
Why this can happenTextData’s response
Seek contrary evidence and preserve reasons for revision.
Records let a researcher inspect stated reasons, actions and revisions. They do not reveal every cause of model behavior.
Inspect the evidence and limitsNo matching modes. Try a different word or show all.
What the components show
Seven evidence summaries connect design responses to demonstrations and remaining limits. They describe bounded results, not a reliability score for the whole system.
Unsupported claimsA documented correction
Design response
Link claims to sources and separate production from review.
What happened
A methods reviewer said that a paper did not identify its evaluators as authors. A fresh reading of the author-contributions section overturned that claim. The corrected qualification was retained in the Grounded Theory toolkit.
What it does not establish
One correction is not an error-rate estimate. The methods inquiry used cross-vendor review, but it was unblinded and did not include an independent second search. Its later synthesis remained provisional.
Explore the component
False confidence or premature completionExplicit limits in retained work
Design response
Retain unknowns and use researcher-defined stopping criteria.
What happened
A citation review declined to certify saturation. A measurement card keeps unresolved fields empty and labels them as unknown; the validator refuses a teaching card offered as a submission.
What it does not establish
These records show a qualified stopping decision and bounded structural checks. They do not establish a general detector of false completion, or that every recorded value is substantively correct.
Explore the component
Faulty interpretationLimits exposed by examples
Design response
Separate quotation checks from judgments of meaning.
What happened
In the annotation prototype, deliberately substituting a consulted party for the responsible actor passes the source checks. So does omitting an assignment. In the synthetic Gioia memo, the absence of a reported action is incorrectly treated as evidence that it did not happen.
What it does not establish
The annotation errors were constructed tests, not observed attempts by a model to game the task. The Gioia example is synthetic. Quotation matching alone cannot establish coding accuracy or an interpretation’s warrant.
Explore the component
Premature agreement or lost contextAn interpretation revised
Design response
Seek contrary evidence and preserve reasons for revision.
What happened
An interview account described an agent sending a message without the expected approval, although the speaker liked the result. Review led to a revised record that separated approval of the outcome from authorization to act, while retaining the earlier record.
What it does not establish
The non-producing reviewer used the producers’ model, saw the earlier analysis and examined selected passages. This was not independent methodological validation or a test of reduced sycophancy. Preserving context does not ensure an agent will retrieve it.
Explore the component
Changed scope or unauthorized actionA bounded software guard
Design response
Bind execution to the approved plan.
What happened
The case-method maintenance software binds a move plan to the materials it inspected. In its synthetic exercise, it refuses the plan when those materials change after the plan is sealed.
What it does not establish
The guard applies to that maintenance operation. It is not general control over agent actions, and it does not implement case selection, case analysis or theory development.
Explore the component
Inconsistency and model changeModel evaluation remains proposed
Design response
Preserve inputs; evaluate repeated runs and version changes.
What happened
The annotation prototype retains requests and candidate responses. Its export can be replayed from saved records. The proposed evaluation would generate new judgments on the same passages and compare them across runs and model versions.
What it does not establish
A replay repeats software processing, not model generation. The historical interactive run’s backend model and settings are unknown. Repeated model runs and cross-version comparisons have not been performed.
Explore the component
Prompt injection, bias and leakageAn open evaluation need
Design response
No general safeguards demonstrated.
What happened
The Grounded Theory release withholds interview transcripts. That is a specific distribution decision; it is not an evaluation of how an agent handles malicious source instructions, biased judgments or confidential information.
What it does not establish
The current demonstrations do not establish general safeguards against any of these three risks. They require separate, task-specific evaluation before broader use.
Explore the component
The reasoning behind the design
Read the longer account of these failure modes, how they can arise and which workflow responses remain proposals.
When AI goes wrong in textual research