{"schema_version":"2.0","title":"TextData — Agent Design and Orchestration Database","as_of":"2026-10-29","description":"A database of papers, repositories, specifications and reusable components for agent design and orchestration.","scope":"34 entries organized into seven categories. Entries describe mechanisms, supporting evidence, limitations and possible uses in TextData. Inclusion does not imply endorsement or evaluated TextData performance.","attribution":"Original authors and maintainers are credited through each entry and its linked references.","families":[{"id":"action","name":"Action and planning","question":"How should an agent choose, execute and revise its next step?","count":5},{"id":"orchestration","name":"Orchestration and reuse","question":"Which responsibilities belong in program structure, and which in a model decision?","count":4},{"id":"state","name":"State and context","question":"What must persist, and what does the model need to see now?","count":5},{"id":"interfaces","name":"Tools and skills","question":"How should capabilities and methodological instructions be exposed?","count":3},{"id":"coordination","name":"Delegation and handoff","question":"When does dividing work help, and what must pass between workers?","count":5},{"id":"evaluation","name":"Evaluation and adaptation","question":"How will we assess a change without confusing extra computation with better design?","count":7},{"id":"evidence","name":"Evidence and authority","question":"How can we recover the basis of a claim and keep source content within its proper role?","count":5}],"records":[{"id":"AD-001","name":"CoALA","family":"action","kind":"Conceptual paper","mechanism":"Organizes language agents through memory, available actions and decision cycles.","evidence":"A conceptual framework relates cognitive architectures to language agents and classifies existing systems.","limitations":"The taxonomy organizes mechanisms; it does not test TextData or prescribe its four-element vocabulary.","textdata_use":"Describe what a configured research agent can read, decide and change.","textdata_evaluation":"Can researchers recover the agent’s inputs, available actions and reasons for consequential transitions?","credit":"Sumers, Yao, Narasimhan and Griffiths (2024).","source_version":"2024, arXiv v3","primary_locator":"§§4.1–4.6; Figs. 4–5","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2024, arXiv v3","locator":"§§4.1–4.6; Figs. 4–5","observed_at":"2026-10-29","source":"arxiv.org/html/2309.02427v3"}],"primary_source":"arxiv.org/html/2309.02427v3","repositories":["GitHub: ysymyth/awesome-language-agents"]},{"id":"AD-002","name":"Building effective agents","family":"orchestration","kind":"Engineering account","mechanism":"Distinguishes predefined execution paths from model-directed steps; describes routing, parallelization and orchestrator–worker patterns.","evidence":"First-party engineering guidance illustrates recurring arrangements.","limitations":"Its narrow engineering use of workflow differs from TextData’s methodological use. It is not a controlled comparison of all patterns.","textdata_use":"Select the simplest execution arrangement appropriate to a method and task.","textdata_evaluation":"Does added delegation improve the research output enough to justify its coordination cost?","credit":"Erik Schluntz and Barry Zhang, Anthropic (2024).","source_version":"2024 article, mutable current page","primary_locator":"What are agents?; Orchestrator-workers; Agents","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2024 article, mutable current page","locator":"What are agents?; Orchestrator-workers; Agents","observed_at":"2026-10-29","source":"anthropic.com/research/building-effective-agents"},{"version":"Published2024-12-19; mutable body observed2026-09-20 with newer SDK/Haiku4.5/ManagedAgents note; exact update date unknown","locator":"anthropic.com/engineering/building-effective-agents What are agents; Building blocks; Agents; Appendix 2","observed_at":"2026-09-20T17:51:32.240461+00:00","source":"anthropic.com/engineering/building-effective-agents"}],"primary_source":"anthropic.com/research/building-effective-agents","repositories":[]},{"id":"AD-003","name":"ReAct","family":"action","kind":"Paper and repository","mechanism":"Interleaves model reasoning, tool actions and observations from the environment.","evidence":"The paper demonstrates the mechanism on question answering and interactive tasks.","limitations":"Task and model results do not establish the validity of an organizational interpretation; prompts and feedback affect behavior.","textdata_use":"Let an agent request a source passage or calculation, inspect the returned evidence and revise its next step.","textdata_evaluation":"Does a claim actually change when a retrieved passage contradicts the initial interpretation?","credit":"Yao et al. (2023).","source_version":"2023, arXiv v3","primary_locator":"§2; §3.1 Action Space; §3.2; Appendix A.3","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2023, arXiv v3","locator":"§2; §3.1 Action Space; §3.2; Appendix A.3","observed_at":"2026-10-29","source":"arxiv.org/html/2210.03629v3"}],"primary_source":"arxiv.org/html/2210.03629v3","repositories":["GitHub: ysymyth/ReAct"]},{"id":"AD-004","name":"AutoGen","family":"coordination","kind":"Paper and repository","mechanism":"Configurable agents exchange messages under programmable conversation and termination rules.","evidence":"The inspected paper combines language models, tools and human inputs in several applications.","limitations":"The 2023 paper manifestation is distinct from later software APIs. Agent roles alone do not establish expertise, independent errors or successful collaboration.","textdata_use":"Specify the evidence, work product and completion condition expected from each delegated role.","textdata_evaluation":"Compare the same research task with one agent and with explicitly partitioned work, accounting for calls and context.","credit":"Wu et al. (2024); inspected arXiv v2 (2023).","source_version":"2023 preprint v2; 2024 COLM publication identity","primary_locator":"§§2.1–2.2","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2023 preprint v2; 2024 COLM publication identity","locator":"§§2.1–2.2","observed_at":"2026-10-29","source":"arxiv.org/html/2308.08155v2"},{"version":"COLM2024 venue metadata; title uses Conversations","locator":null,"observed_at":null,"source":"openreview.net/forum?id=BAakY1hNKS"}],"primary_source":"arxiv.org/html/2308.08155v2","repositories":["GitHub: microsoft/autogen"]},{"id":"AD-005","name":"DSPy","family":"orchestration","kind":"Paper and repository","mechanism":"Separates declared input/output interfaces and modular programs from prompts optimized against a metric.","evidence":"The paper evaluates compilation of language-model programs on specified question-answering and reasoning tasks.","limitations":"Optimization is tied to data, model and metric. Interface reuse does not guarantee stable scientific meaning after a model change.","textdata_use":"Keep a construct specification separate from the prompt implementation used to extract or classify evidence.","textdata_evaluation":"Compare unchanged reuse, renewed optimization and a manual baseline on held-out materials.","credit":"Khattab et al. (2024); inspected 2023 arXiv manifestation.","source_version":"2023 preprint v1; 2024 ICLR publication identity","primary_locator":"§§3.1–3.3; §4","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2023 preprint v1; 2024 ICLR publication identity","locator":"§§3.1–3.3; §4","observed_at":"2026-10-29","source":"arxiv.org/html/2310.03714v1"},{"version":"ICLR2024","locator":"openreview.net/notes/edits/attachment?id=wU1R77dqOm&name=pdf ; pp1,6–9,14","observed_at":null,"source":"openreview.net/notes/edits/attachment?id=wU1R77dqOm&name=pdf"},{"version":"3.3.1 / 3.4.0b1","locator":"GitHub: stanfordnlp/dspy/releases ; 3.4.0 Beta1 compatibility and migration;3.3.1","observed_at":null,"source":"GitHub: stanfordnlp/dspy/releases"},{"version":"mutable main","locator":"GitHub: stanfordnlp/dspy/blob/main/docs/docs/tutorials/saving/index.md ; State-only, Whole Program, Backward Compatibility","observed_at":null,"source":"GitHub: stanfordnlp/dspy/blob/main/docs/docs/tutorials/saving/index.md"},{"version":"mutable main","locator":"GitHub: stanfordnlp/dspy/blob/main/docs/docs/learn/optimization/optimizers.md ; What an optimizer tunes;Which optimizer","observed_at":null,"source":"GitHub: stanfordnlp/dspy/blob/main/docs/docs/learn/optimization/optimizers.md"}],"primary_source":"arxiv.org/html/2310.03714v1","repositories":["GitHub: stanfordnlp/dspy"]},{"id":"AD-006","name":"Model Context Protocol resources and tools","family":"interfaces","kind":"Specification","mechanism":"Resources expose contextual data; tools expose callable operations through a shared protocol.","evidence":"The current resource specification explicitly describes contextual data and application-controlled presentation.","limitations":"Protocol compatibility does not establish source validity, permission to act, transactional effects or implementation equivalence.","textdata_use":"Expose source collections and analytic operations as distinct interfaces without conflating them with study design.","textdata_evaluation":"Can a tool call be traced to its input source and study-specific assumptions?","credit":"Model Context Protocol contributors.","source_version":"2026-07-28 specification; 2025-06-18 historical comparison","primary_locator":"Resources and Tools: introductions and user interaction models","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2026-07-28 specification; 2025-06-18 historical comparison","locator":"Resources and Tools: introductions and user interaction models","observed_at":"2026-10-29","source":"modelcontextprotocol.io/specification/2026-07-28/server/resources"}],"primary_source":"modelcontextprotocol.io/specification/2026-07-28/server/resources","repositories":["GitHub: modelcontextprotocol/specification"]},{"id":"AD-007","name":"Co-Scientist","family":"coordination","kind":"Scientific system paper","mechanism":"Converts a scientist’s goal and constraints into a plan with specialized generation, critique and refinement work.","evidence":"The Nature article reports biomedical hypothesis development with expert assessment and experimental follow-up.","limitations":"The demonstrations are biomedical and include human/laboratory contributions. Co-Scientist is distinct from the chemistry system Coscientist; no public implementation is inferred.","textdata_use":"Organize a bounded research task around a question, admissible evidence and requested output.","textdata_evaluation":"Does coordination yield better-supported rival explanations on organizational texts?","credit":"Gottweis et al. (2026).","source_version":"2026 published Nature article","primary_locator":"Fig. 1; Methods: From research goal to research plan configuration; The specialized agents underpinning Co-Scientist; limitations discussion","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2026 published Nature article","locator":"Fig. 1; Methods: From research goal to research plan configuration; The specialized agents underpinning Co-Scientist; limitations discussion","observed_at":"2026-10-29","source":"nature.com/articles/s41586-026-10644-y"}],"primary_source":"nature.com/articles/s41586-026-10644-y","repositories":[]},{"id":"AD-008","name":"Anthropic multi-agent Research","family":"coordination","kind":"Engineering account","mechanism":"A lead agent delegates bounded searches, receives findings and decides whether to continue.","evidence":"The first-party account describes deployed design decisions and internal evaluation.","limitations":"Larger resource use and internal tasks prevent isolating agent-count benefits or assuming transfer to scholarly review.","textdata_use":"Partition independent search questions while retaining source references for synthesis.","textdata_evaluation":"Measure missed evidence, overlap and integration effort against a comparable single-agent search.","credit":"Anthropic engineering contributors (2025).","source_version":"2025-06-13, current mutable page","primary_locator":"Architecture overview; delegation principle 2; Benefits; Effective evaluation","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2025-06-13, current mutable page","locator":"Architecture overview; delegation principle 2; Benefits; Effective evaluation","observed_at":"2026-10-29","source":"anthropic.com/research/multi-agent-research-system"}],"primary_source":"anthropic.com/research/multi-agent-research-system","repositories":[]},{"id":"AD-009","name":"Effective context engineering","family":"state","kind":"Engineering account","mechanism":"Uses references, selective retrieval and compaction to manage what a model sees during long tasks.","evidence":"First-party guidance explains context selection and long-horizon working practices.","limitations":"Compaction can omit distinctions; smaller context is not necessarily more faithful context.","textdata_use":"Load relevant passages and method guidance while retaining separately addressable originals.","textdata_evaluation":"Test whether a resumed analysis retains contrary evidence and consequential study commitments.","credit":"Anthropic engineering contributors (2025).","source_version":"2025-09-29, current mutable page","primary_locator":"Context retrieval and agentic search; long-horizon context/compaction","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2025-09-29, current mutable page","locator":"Context retrieval and agentic search; long-horizon context/compaction","observed_at":"2026-10-29","source":"anthropic.com/research/effective-context-engineering-for-ai-agents"}],"primary_source":"anthropic.com/research/effective-context-engineering-for-ai-agents","repositories":[]},{"id":"AD-010","name":"LLooM","family":"evidence","kind":"Paper and repository","mechanism":"Supports concept induction with analyst inspection, editing and re-analysis.","evidence":"The paper combines concept-level evaluations with exploratory expert cases.","limitations":"The authors identify overreliance and bias risks. Concept scores and editable concepts do not by themselves validate an organizational explanation.","textdata_use":"Keep generated concepts open to source inspection and researcher revision.","textdata_evaluation":"Do analysts detect missed concepts and unsupported distinctions, including alternatives outside the generated set?","credit":"Lam, Teoh, Landay, Heer and Bernstein (2024).","source_version":"2024, CHI article / arXiv v1","primary_locator":"§3.2.2; §6 opening and scope; §7.2.3","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2024, CHI article / arXiv v1","locator":"§3.2.2; §6 opening and scope; §7.2.3","observed_at":"2026-10-29","source":"arxiv.org/html/2404.12259v1"}],"primary_source":"arxiv.org/html/2404.12259v1","repositories":["GitHub: stanfordhci/lloom"]},{"id":"AD-011","name":"MAST","family":"coordination","kind":"Failure-analysis paper","mechanism":"Classifies system-design, inter-agent alignment and task-verification failures.","evidence":"The primary paper supplies failure categories and selected intervention studies.","limitations":"The identities and correspondence of released traces and annotations remain unresolved in the documented evidence. Use the taxonomy provisionally; dataset counts should not be treated as independent observations or as a universal failure rate.","textdata_use":"Inspect whether failures arise in the task specification, handoff or assessment of completion.","textdata_evaluation":"Can the failure classification distinguish missing evidence from incorrect synthesis and premature stopping?","credit":"Cemri et al. (2025).","source_version":"2025, arXiv v3; NeurIPS proceedings identity","primary_locator":"§§3–5; Appendix A failure categories","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2025, arXiv v3; NeurIPS proceedings identity","locator":"§§3–5; Appendix A failure categories","observed_at":"2026-10-29","source":"arxiv.org/html/2503.13657v3"},{"version":"2026-05-06","locator":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/13 Description / reproduction / qualification","observed_at":null,"source":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/13"},{"version":"2026-08-23","locator":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/18 Description / reproduction / qualification","observed_at":null,"source":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/18"},{"version":"2026-08-09","locator":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/17 Description / reproduction / qualification","observed_at":null,"source":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/17"},{"version":"2026-08-07","locator":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/16 Description / reproduction / qualification","observed_at":null,"source":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/16"},{"version":"2026-05-06","locator":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/12 Description / reproduction / qualification","observed_at":null,"source":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/12"},{"version":"2025-10-30 issue; mutable","locator":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/10 Description / reproduction and limitations","observed_at":null,"source":"GitHub: multi-agent-systems-failure-taxonomy/MAST/issues/10"},{"version":"95118ac951421753cf1deb87ddea3b01e693c41b at2026-08-13T19:46:12Z; prior6bd601d98498910e079b4fd0a7a6d4c0ab3cae60 at19:38:25Z","locator":"huggingface.co/datasets/mcemri/MAST-Data/commit/95118ac951421753cf1deb87ddea3b01e693c41b; dataset commit metadata and pinned README, Notes","observed_at":"2026-09-21T02:55:22.055473+00:00","source":"huggingface.co/datasets/mcemri/MAST-Data/commit/95118ac951421753cf1deb87ddea3b01e693c41b"}],"primary_source":"arxiv.org/html/2503.13657v3","repositories":["GitHub: multi-agent-systems-failure-taxonomy/MAST"]},{"id":"AD-012","name":"W3C PROV-DM","family":"evidence","kind":"Specification","mechanism":"Represents entities, activities, derivation and responsibility.","evidence":"A domain-independent provenance model specifies how outputs relate to inputs and activities.","limitations":"Provenance describes derivation, not truth. A PROV agent can be a responsible person or system and is not synonymous with an LLM agent.","textdata_use":"Connect source passages, derived codes, memos and claims without treating their derivation as validation.","textdata_evaluation":"Can a reader trace an interpretation to the source version and decision that produced it?","credit":"Moreau, Missier and W3C Provenance Working Group (2013).","source_version":"W3C Recommendation 2013-04-30","primary_locator":"§§2.1, 2.2.2, 4, 5.2–5.3","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"W3C Recommendation 2013-04-30","locator":"§§2.1, 2.2.2, 4, 5.2–5.3","observed_at":"2026-10-29","source":"w3.org/TR/2013/REC-prov-dm-20130430/"}],"primary_source":"w3.org/TR/2013/REC-prov-dm-20130430/","repositories":[]},{"id":"AD-013","name":"Managed Agents architecture","family":"state","kind":"Engineering account","mechanism":"Separates the durable session event log, the agent execution loop and the sandbox.","evidence":"The April 2026 engineering account explains these interfaces and how the current model context is assembled.","limitations":"A vendor architecture account is not independent evidence of TextData reliability. A durable log does not guarantee complete or correct model context.","textdata_use":"Retain study records outside a replaceable agent session while making the current context inspectable.","textdata_evaluation":"After replacing a model or worker, can the inquiry resume with its evidence and unresolved decisions intact?","credit":"Anthropic engineering contributors (2026).","source_version":"2026-04-08, current mutable page","primary_locator":"The session is not Claude’s context window; stable interfaces","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2026-04-08, current mutable page","locator":"The session is not Claude’s context window; stable interfaces","observed_at":"2026-10-29","source":"anthropic.com/research/managed-agents"},{"version":"2026-04-08","locator":"anthropic.com/engineering/managed-agents ; Decoupling architecture","observed_at":null,"source":"anthropic.com/engineering/managed-agents"},{"version":"beta header2026-04-01","locator":"platform.claude.com/docs/en/managed-agents/overview ; Core concepts;Beta access","observed_at":null,"source":"platform.claude.com/docs/en/managed-agents/overview"}],"primary_source":"anthropic.com/research/managed-agents","repositories":[]},{"id":"AD-014","name":"LangGraph persistence","family":"state","kind":"Repository and documentation","mechanism":"Separates thread-state checkpoints from cross-thread application data stored in a store.","evidence":"Official documentation distinguishes the two interfaces and warns that in-memory savers do not survive process restart.","limitations":"Restart durability depends on the backing store and configuration. Checkpoints do not make arbitrary external effects atomic or establish analysis validity.","textdata_use":"Keep execution position separate from the enduring study evidence and decisions.","textdata_evaluation":"Interrupt a task and check both correct resumption and absence of duplicated external effects.","credit":"Original authors and maintainers are identified in the linked publication or repository.","source_version":"current documentation, accessed 2026-10-29","primary_locator":"Overview; Checkpointer vs. store","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"current documentation, accessed 2026-10-29","locator":"Overview; Checkpointer vs. store","observed_at":"2026-10-29","source":"docs.langchain.com/oss/python/langgraph/persistence"},{"version":"1.2.11","locator":null,"observed_at":null,"source":"GitHub: langchain-ai/langgraph/releases/tag/1.2.11"},{"version":"mutable docs","locator":"docs.langchain.com/oss/python/langgraph/graph-api ; Graph migrations","observed_at":null,"source":"docs.langchain.com/oss/python/langgraph/graph-api"},{"version":"Sep6,2026","locator":"GitHub: langchain-ai/langgraph/issues/8834 ; Repro;Observed output;Environment","observed_at":null,"source":"GitHub: langchain-ai/langgraph/issues/8834"},{"version":"Sep2026;SQLite3.1.1","locator":"GitHub: langchain-ai/langgraph/issues/8833 ; Observed output,System Info","observed_at":null,"source":"GitHub: langchain-ai/langgraph/issues/8833"},{"version":"Dec9,2025 advisory","locator":"GitHub: langchain-ai/langgraph/security/advisories/GHSA-9rwj-6rc7-p77c ; Affected/Patched versions","observed_at":null,"source":"GitHub: langchain-ai/langgraph/security/advisories/GHSA-9rwj-6rc7-p77c"},{"version":"1.1.3–1.1.6;langgraph-api0.7.28/0.7.96;Apr5,2026","locator":"GitHub: langchain-ai/langgraph/issues/7417 ; issue body","observed_at":null,"source":"GitHub: langchain-ai/langgraph/issues/7417"},{"version":"1.1.7 through1.2.9;Jul27,2026","locator":"GitHub: langchain-ai/langgraph/issues/8458 ; issue body","observed_at":null,"source":"GitHub: langchain-ai/langgraph/issues/8458"},{"version":"Aug9,2026 main version not pinned","locator":"GitHub: langchain-ai/langgraph/issues/8579 ; issue body","observed_at":null,"source":"GitHub: langchain-ai/langgraph/issues/8579"},{"version":"June11,2026","locator":"research.checkpoint.com/2026/from-sqli-to-rce-exploiting-langgraphs-checkpointer/ ; Key Points / Disclosure Timeline","observed_at":null,"source":"research.checkpoint.com/2026/from-sqli-to-rce-exploiting-langgraphs-checkpointer/"},{"version":"Mutable documentation; observed 20 September 2026.","locator":"docs.langchain.com/oss/python/langgraph/checkpointers ; Checkpoints / pending writes / Durability modes","observed_at":"2026-09-20T19:40:58.232Z","source":"docs.langchain.com/oss/python/langgraph/checkpointers"}],"primary_source":"docs.langchain.com/oss/python/langgraph/persistence","repositories":["GitHub: langchain-ai/langgraph"]},{"id":"AD-015","name":"Reflexion","family":"action","kind":"Paper and repository","mechanism":"Stores verbal feedback from prior attempts and uses it to guide a later attempt without updating model weights.","evidence":"The paper extraction includes task improvements and harmful-reflection ablations.","limitations":"Self-generated feedback can reinforce mistakes; more attempts and useful external feedback affect comparisons.","textdata_use":"Turn a documented error into an explicit revision question while preserving the previous interpretation.","textdata_evaluation":"Does revision repair the identified source mismatch without introducing new unsupported claims?","credit":"Shinn et al. (2023).","source_version":"v4 2023-10-10","primary_locator":"arxiv.org/html/2303.11366v4 Sections 3, 4.1–4.3; Table 3; Section 5","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"v4 2023-10-10","locator":"arxiv.org/html/2303.11366v4 Sections 3, 4.1–4.3; Table 3; Section 5","observed_at":"2026-09-20T17:51:32.242164+00:00","source":"arxiv.org/html/2303.11366v4"}],"primary_source":"arxiv.org/html/2303.11366v4","repositories":["GitHub: noahshinn/reflexion"]},{"id":"AD-016","name":"Tree of Thoughts","family":"action","kind":"Paper and repository","mechanism":"Searches alternative intermediate states using model-based evaluation and tree search.","evidence":"The paper describes breadth-first and depth-first search on bounded reasoning tasks.","limitations":"Benefits depend on useful candidate generation and evaluation, and on added computation. A coherent path need not be empirically supported.","textdata_use":"Explore rival analytic interpretations before retaining a category or explanation.","textdata_evaluation":"Compare alternatives with equal-budget sampling and evaluate them against source evidence.","credit":"Yao et al. (2023).","source_version":"v2 2023-12-03; NeurIPS camera ready","primary_locator":"arxiv.org/html/2305.10601v2 Section 4.1 Table 2; Section 6; Appendix B.3 Table 7","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"v2 2023-12-03; NeurIPS camera ready","locator":"arxiv.org/html/2305.10601v2 Section 4.1 Table 2; Section 6; Appendix B.3 Table 7","observed_at":"2026-09-20T17:51:32.242454+00:00","source":"arxiv.org/html/2305.10601v2"}],"primary_source":"arxiv.org/html/2305.10601v2","repositories":["GitHub: princeton-nlp/tree-of-thought-llm"]},{"id":"AD-017","name":"LLMCompiler","family":"action","kind":"Paper and repository","mechanism":"Plans tool calls with dependencies, dispatches independent calls in parallel and can replan after observations.","evidence":"The paper evaluates function-calling workloads and interactive tasks; parallelism helps where the dependency structure permits it.","limitations":"Sequential interactive tasks can incur planning overhead. Reported speedups do not establish faster or better organizational research.","textdata_use":"Parallelize independent document preparation or retrieval while preserving dependent interpretation steps.","textdata_evaluation":"Compare result fidelity and elapsed time with sequential execution on the same task graph.","credit":"Kim et al. (2024).","source_version":"2024-06-05 v3","primary_locator":"§3; §5.4; Appendix B","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2024-06-05 v3","locator":"§3; §5.4; Appendix B","observed_at":"2026-10-29","source":"arxiv.org/html/2312.04511v3"},{"version":"2024-06-05 v3","locator":"arxiv.org/html/2312.04511v3 §3–4,Tables1–3,AppendixA/D","observed_at":"2026-09-20T18:49:31.746734+00:00","source":"arxiv.org/html/2312.04511v3"}],"primary_source":"arxiv.org/html/2312.04511v3","repositories":["GitHub: SqueezeAILab/LLMCompiler"]},{"id":"AD-018","name":"Open Agent Specification","family":"orchestration","kind":"Paper and repository","mechanism":"Serializes agent and workflow definitions and connects them to runtime adapters.","evidence":"The empirical paper runs related configurations across frameworks and reports differences in outputs and latency.","limitations":"An older technical report deferred evaluation; the later empirical manifestation supersedes that claim. Configuration interchange does not demonstrate suspended-state migration or identical semantics.","textdata_use":"Separate an explicit study/task specification from a particular execution framework.","textdata_evaluation":"Run the same specified operation across implementations and assess semantic, output and recovery differences.","credit":"Agent Spec paper authors and Oracle contributors.","source_version":"Empirical arXiv v4 (2025), CAIS 2026 identity; software 26.3.1 documented in September 2026.","primary_locator":"arxiv.org/html/2510.04173v4 ; §5.1–5.2 Tables3–5","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"v4 Nov7,2025;older v1 technical report","locator":"arxiv.org/html/2510.04173v4 ; §5.1–5.2 Tables3–5","observed_at":null,"source":"arxiv.org/html/2510.04173v4"},{"version":"CAIS2026 May26","locator":"doi.org/10.1145/3786335.3813130 ; §2.2;§4 tool semantics;§5.1","observed_at":null,"source":"doi.org/10.1145/3786335.3813130"},{"version":"report references25.4.0;mutable development URL","locator":"oracle.github.io/agent-spec/development/_downloads/9bed8c437b5139f7c816476b7e2c20f1/agentspec_technical_report.pdf ; §2.3–2.6,especially§2.6.1","observed_at":null,"source":"oracle.github.io/agent-spec/development/_downloads/9bed8c437b5139f7c816476b7e2c20f1/agentspec_technical_report.pdf"},{"version":"26.3.1 Sep11,2026","locator":"GitHub: oracle/agent-spec/releases ; 26.3.1 Dependencies,LangGraph improvements,Breaking changes","observed_at":null,"source":"GitHub: oracle/agent-spec/releases"},{"version":"26.3.1","locator":"oracle.github.io/agent-spec/26.3.1/howtoguides/howto_execute_agentspec_across_frameworks.html ; Adapters;tool_registry;license comment","observed_at":null,"source":"oracle.github.io/agent-spec/26.3.1/howtoguides/howto_execute_agentspec_across_frameworks.html"},{"version":"Jan20,2026;open observed","locator":"GitHub: oracle/agent-spec/issues/81 ; Context/Acceptance","observed_at":null,"source":"GitHub: oracle/agent-spec/issues/81"}],"primary_source":"arxiv.org/html/2510.04173v4","repositories":["GitHub: oracle/agent-spec"]},{"id":"AD-019","name":"Temporal agent integration","family":"orchestration","kind":"Repository and documentation","mechanism":"Runs an agent loop inside a durable workflow and records model and I/O operations as replayable activities.","evidence":"The versioned integration README documents replay, session carryover and implementation restrictions.","limitations":"Inspected integration 1.24.0 is experimental. Replaying recorded results does not guarantee exactly-once remote effects; continuation and approval support have explicit limits.","textdata_use":"Explore recovery for long research operations whose dependencies and external effects are explicit.","textdata_evaluation":"Check interrupted writes, retries, session continuity and effects separately before relying on recovery.","credit":"Temporal contributors.","source_version":"\u0040temporalio/openai-agents1.24.0 experimental; npm date verified","primary_locator":"GitHub: temporalio/sdk-typescript/blob/v1.24.0/contrib/openai-agents/README.md ; Introduction; Tools; modelParams","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"\u0040temporalio/openai-agents1.24.0 experimental; npm date verified","locator":"GitHub: temporalio/sdk-typescript/blob/v1.24.0/contrib/openai-agents/README.md ; Introduction; Tools; modelParams","observed_at":null,"source":"GitHub: temporalio/sdk-typescript/blob/v1.24.0/contrib/openai-agents/README.md"}],"primary_source":"GitHub: temporalio/sdk-typescript/blob/v1.24.0/contrib/openai-agents/README.md","repositories":["GitHub: temporalio/sdk-typescript"]},{"id":"AD-020","name":"MemGPT","family":"state","kind":"Paper and repository","mechanism":"Moves information between limited working context and external recall/archival storage through explicit function calls.","evidence":"The paper evaluates conversational memory and document analysis and describes lossy recursive summaries beside retained history.","limitations":"A stored source may still be missed or poorly summarized. Historical MemGPT results are not guarantees for the current Letta product.","textdata_use":"Keep original study materials recoverable when working summaries are compacted.","textdata_evaluation":"Test recovery of a rarely used contrary passage after several analysis cycles.","credit":"Packer et al.; inspected 2024 arXiv v2.","source_version":"2024-02-12 v2","primary_locator":"§2.1–2.4; §3","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2024-02-12 v2","locator":"§2.1–2.4; §3","observed_at":"2026-10-29","source":"arxiv.org/html/2310.08560v2"},{"version":"2024-02-12 v2","locator":"arxiv.org/html/2310.08560v2 — §2, §3.1, §3.2.1","observed_at":null,"source":"arxiv.org/html/2310.08560v2"}],"primary_source":"arxiv.org/html/2310.08560v2","repositories":["GitHub: letta-ai/letta"]},{"id":"AD-021","name":"Mem0","family":"state","kind":"Paper and repository","mechanism":"Extracts and retrieves persistent information; the inspected paper distinguishes extraction from update decisions.","evidence":"The paper and product records describe different memory representations and operating contracts.","limitations":"The 2025 paper’s update/graph operations cannot be assumed for later OSS or hosted products. Benchmark budgets and representations differ; no present product winner is inferred.","textdata_use":"Treat an extracted fact as a versioned interpretation linked to the passage it summarizes.","textdata_evaluation":"Test stale facts, contradictions and source recovery rather than only successful recall.","credit":"Chhikara et al. (2025) and Mem0 maintainers.","source_version":"2025-04-28 v1","primary_locator":"arxiv.org/html/2504.19413v1 — §2.1–2.2; §3.1","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2025-04-28 v1","locator":"arxiv.org/html/2504.19413v1 — §2.1–2.2; §3.1","observed_at":null,"source":"arxiv.org/html/2504.19413v1"},{"version":"cutoff commita39a802bbc93e85b820078cd3c4dbaf53af25dbe","locator":null,"observed_at":null,"source":"GitHub: mem0ai/mem0"},{"version":"2026-08-01","locator":"GitHub: mem0ai/mem0/releases/tag/v2.0.15 — Bug Fixes Core","observed_at":null,"source":"GitHub: mem0ai/mem0/releases/tag/v2.0.15"},{"version":"mutable; exact shipment date unresolved","locator":"docs.mem0.ai/migration/oss-v2-to-v3 — Overview, Breaking Changes, Entity collection","observed_at":null,"source":"docs.mem0.ai/migration/oss-v2-to-v3"},{"version":"mutable observed after cutoff; precise revision unknown","locator":"docs.aws.amazon.com/AmazonElastiCache/latest/dg/agentic-memory-best-practices.html — lifecycle best practices","observed_at":null,"source":"docs.aws.amazon.com/AmazonElastiCache/latest/dg/agentic-memory-best-practices.html"}],"primary_source":"arxiv.org/html/2504.19413v1","repositories":["GitHub: mem0ai/mem0"]},{"id":"AD-022","name":"Agent Skills","family":"interfaces","kind":"Specification and repository","mechanism":"Packages instructions with optional scripts, references and assets, with progressive loading of detail.","evidence":"The official format specifies SKILL.md metadata and instructions, with optional scripts, references and assets.","limitations":"A valid package does not establish methodological correctness or identical client behavior. The allowed-tools field is experimental.","textdata_use":"Package method-specific research procedures while retaining their authorship, examples and scope.","textdata_evaluation":"Can another researcher use the same procedure and identify its methodological assumptions and boundaries?","credit":"Agent Skills specification contributors.","source_version":"Mutable official specification inspected 2026-10-29; no package release tested","primary_locator":"Directory structure; SKILL.md; progressive disclosure","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"Mutable official specification inspected 2026-10-29; no package release tested","locator":"Directory structure; SKILL.md; progressive disclosure","observed_at":"2026-10-29","source":"agentskills.io/specification"},{"version":"Version not recorded.","locator":"agentskills.io/specification :: #directory-structure; #frontmatter; #compatibility-field; #allowed-tools-field","observed_at":null,"source":"agentskills.io/specification"},{"version":"Version not recorded.","locator":"agentskills.io/client-implementation/adding-skills-support :: #where-to-scan; #handling-name-collisions; #trust-considerations; #filtering; #permission-allowlisting","observed_at":null,"source":"agentskills.io/client-implementation/adding-skills-support"},{"version":"Version not recorded.","locator":null,"observed_at":null,"source":"GitHub: agentskills/agentskills"}],"primary_source":"agentskills.io/specification","repositories":["GitHub: agentskills/agentskills"]},{"id":"AD-023","name":"SWE-agent","family":"interfaces","kind":"Paper and repository","mechanism":"Designs the agent–computer interface around search, a bounded document view, editing and feedback.","evidence":"The paper’s interface ablations test effects on software-engineering tasks with a specified model.","limitations":"Coding benchmarks do not validate text interpretation. The need for each interface feature can change with the model.","textdata_use":"Make passage inspection and analytic operations easy to invoke and inspect.","textdata_evaluation":"Compare a generic tool interface with a task-specific one for missed context and unsupported edits.","credit":"Yang et al. (2024).","source_version":"Version not recorded.","primary_locator":"arxiv.org/html/2405.15793v3 | Sections 3, 4, 5.1; lines 103-193","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"Version not recorded.","locator":"arxiv.org/html/2405.15793v3 | Sections 3, 4, 5.1; lines 103-193","observed_at":null,"source":"arxiv.org/html/2405.15793v3"}],"primary_source":"arxiv.org/html/2405.15793v3","repositories":["GitHub: SWE-agent/SWE-agent"]},{"id":"AD-024","name":"Who Validates the Validators / EvalGen","family":"evaluation","kind":"Paper and research tool","mechanism":"Develops evaluators through user inspection and revision of criteria.","evidence":"The qualitative and empirical study shows that evaluation criteria may change as people inspect outputs.","limitations":"Matching an earlier rubric does not establish a permanent gold standard. Human inspection itself remains situated and fallible.","textdata_use":"Develop task-specific assessment criteria with organizational researchers and retain meaningful revisions.","textdata_evaluation":"Do evaluation criteria capture source fidelity and theoretical usefulness, including disputed cases?","credit":"Shankar et al. (2024).","source_version":"2024 primary manuscript","primary_locator":"arxiv.org/html/2404.12272 — user study and discussion","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2024 primary manuscript","locator":"arxiv.org/html/2404.12272 — user study and discussion","observed_at":null,"source":"arxiv.org/html/2404.12272"}],"primary_source":"arxiv.org/html/2404.12272","repositories":[]},{"id":"AD-025","name":"Inspect AI","family":"evaluation","kind":"Repository and documentation","mechanism":"Connects evaluation tasks, solvers, scorers and inspectable per-run/per-sample logs.","evidence":"Official documentation describes log storage, transcript analysis and exported run configuration. Multiple scorers and repeated evaluations are also documented.","limitations":"The framework executes a chosen evaluation; it does not establish that its scorer measures the intended construct.","textdata_use":"Keep an assessment run separate from the workflow under assessment and preserve its configuration and evidence.","textdata_evaluation":"Can researchers reproduce the assessed materials and explain why each score was assigned?","credit":"UK AI Security Institute and Inspect contributors.","source_version":"Mutable official documentation inspected 2026-10-29; no installed version","primary_locator":"Overview; Log Format; EvalLog; Exporting Run Config","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"Mutable official documentation inspected 2026-10-29; no installed version","locator":"Overview; Log Format; EvalLog; Exporting Run Config","observed_at":"2026-10-29","source":"inspect.aisi.org.uk/eval-logs.html"},{"version":"Version not recorded.","locator":null,"observed_at":"2026-07-30T16:17:45Z","source":"GitHub: UKGovernmentBEIS/inspect_ai"},{"version":"Version not recorded.","locator":null,"observed_at":"2026-07-30T16:17:45Z","source":"inspect.aisi.org.uk"},{"version":"Version not recorded.","locator":"inspect.aisi.org.uk/model-graded.html (Multiple Models and Grading Robustness); inspect.aisi.org.uk/metrics.html (Reducing Epochs); inspect.aisi.org.uk/eval-logs.html (log schema and export-config)","observed_at":"2026-07-30T16:17:45Z","source":"inspect.aisi.org.uk/model-graded.html"},{"version":"Version not recorded.","locator":"inspect.aisi.org.uk/model-graded.html (Multiple Models and Grading Robustness); inspect.aisi.org.uk/metrics.html (Reducing Epochs); inspect.aisi.org.uk/eval-logs.html (log schema and export-config)","observed_at":"2026-07-30T16:17:45Z","source":"inspect.aisi.org.uk/metrics.html"},{"version":"Version not recorded.","locator":"inspect.aisi.org.uk/model-graded.html (Multiple Models and Grading Robustness); inspect.aisi.org.uk/metrics.html (Reducing Epochs); inspect.aisi.org.uk/eval-logs.html (log schema and export-config)","observed_at":"2026-07-30T16:17:45Z","source":"inspect.aisi.org.uk/eval-logs.html"},{"version":"Version not recorded.","locator":null,"observed_at":"2026-07-30T16:17:45Z","source":"inspect.aisi.org.uk/multiple-scorers.html"}],"primary_source":"inspect.aisi.org.uk/eval-logs.html","repositories":["GitHub: UKGovernmentBEIS/inspect_ai"]},{"id":"AD-026","name":"OpenTelemetry GenAI conventions","family":"evidence","kind":"Specification and repository","mechanism":"Defines shared names and relationships for records of model and agent operations.","evidence":"The official documentation directs readers to a separate repository for GenAI semantic conventions covering model and agent operations.","limitations":"The former documentation location is no longer maintained. Trace conventions do not supply source-to-claim reasoning or certify complete instrumentation.","textdata_use":"Connect model/tool activity records to study evidence while retaining their distinct meanings.","textdata_evaluation":"Can an activity log locate the actual source and output artifacts needed to inspect an interpretation?","credit":"OpenTelemetry contributors.","source_version":"Official documentation and repository location checked on 29 October 2026; no software release evaluated.","primary_locator":"Official documentation relocation notice and repository","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Selected primary text","reading_scope":"The entry draws on selected original text; this does not establish that every referenced work was read in full.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"Repository identity; documentation relocation checked 2026-10-29","locator":"Official documentation relocation notice and repository","observed_at":"2026-10-29","source":"GitHub: open-telemetry/semantic-conventions-genai"}],"primary_source":"GitHub: open-telemetry/semantic-conventions-genai","repositories":["GitHub: open-telemetry/semantic-conventions-genai"]},{"id":"AD-027","name":"AgentDojo","family":"evidence","kind":"Benchmark and repository","mechanism":"Combines legitimate tool tasks with attacks through untrusted content and evaluates utility and attack outcomes.","evidence":"The primary project documentation describes a dynamic evaluation environment with task suites, attacks and defenses.","limitations":"Benchmark outcomes depend on attack access, tasks and defenses. No deployment-specific risk rate or complete defense follows.","textdata_use":"Keep retrieved text as evidence rather than allowing embedded instructions to silently redirect an inquiry.","textdata_evaluation":"Can an agent use a source passage while rejecting unrelated instructions carried in that passage?","credit":"AgentDojo authors and ETH Zürich SPY Lab contributors.","source_version":"Version not recorded.","primary_locator":"Project homepage, overview paragraphs and documentation links for tasks, attacks, and defenses","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"Version not recorded.","locator":null,"observed_at":"2026-07-31T05:05:00Z","source":"agentdojo.spylab.ai"},{"version":"Version not recorded.","locator":null,"observed_at":"2026-07-31T05:06:00Z","source":"GitHub: ethz-spylab/agentdojo"}],"primary_source":"agentdojo.spylab.ai","repositories":["GitHub: ethz-spylab/agentdojo"]},{"id":"AD-028","name":"PrivacyLens","family":"evidence","kind":"Benchmark paper","mechanism":"Examines privacy-relevant behavior in agent actions as well as responses about privacy norms.","evidence":"The paper reports a distinction between stated privacy judgments and information exposed during actions.","limitations":"Its tasks and leakage definitions bound the result; probing a norm and observing an action are different measurements.","textdata_use":"Assess how a research workflow handles sensitive contextual information during tool use and output preparation.","textdata_evaluation":"Does the actual exported analysis respect the study’s authorized source and participant boundaries?","credit":"Shao et al.; NeurIPS 2024 identity, inspected arXiv v3 (2025).","source_version":"2025-03-14 v3; NeurIPS 2024","primary_locator":"arxiv.org/html/2409.00138v3 §3–6, Table 2","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"2025-03-14 v3; NeurIPS 2024","locator":"arxiv.org/html/2409.00138v3 §3–6, Table 2","observed_at":"2026-09-20T17:53:16.634879+00:00","source":"arxiv.org/html/2409.00138v3"}],"primary_source":"arxiv.org/html/2409.00138v3","repositories":[]},{"id":"AD-029","name":"Berkeley Function Calling Leaderboard","family":"evaluation","kind":"Benchmark and documentation","mechanism":"Assesses function calling, including state and viable call sequences in multi-turn tasks.","evidence":"The official articles distinguish evaluation categories and requirements beyond well-formed syntax.","limitations":"The documented article weights and live aggregates differ. Historical versions do not establish a single current model ranking.","textdata_use":"Assess whether a requested analytic operation is selected and parameterized correctly.","textdata_evaluation":"Distinguish a valid call, a correct state transition and a substantively appropriate research operation.","credit":"Berkeley Function Calling Leaderboard authors and maintainers.","source_version":"V4 explanatory article; not the live scorer commit","primary_locator":"Full Leaderboard Score Composition table; dataset counts","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"V4 explanatory article; not the live scorer commit","locator":null,"observed_at":null,"source":"gorilla.cs.berkeley.edu/blogs/15_bfcl_v4_web_search.html"},{"version":"V3 multi-turn explanatory article","locator":null,"observed_at":null,"source":"gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html"},{"version":"live page says last updated 2026-04-12; bfcl-eval2025.12.17, commit f7cf735","locator":null,"observed_at":null,"source":"gorilla.cs.berkeley.edu/leaderboard.html"},{"version":"Fanjia Yan, A Function Calling Perspective on Scalable Large Language Model Agent Evaluation; UCB/EECS-2025-184 May14 2025 cover, inner copyright2026/2025 mismatch; not asserted identical to ICML paper","locator":null,"observed_at":null,"source":"www2.eecs.berkeley.edu/Pubs/TechRpts/2025/Archive/EECS-2025-184.pdf"}],"primary_source":"gorilla.cs.berkeley.edu/blogs/15_bfcl_v4_web_search.html","repositories":[]},{"id":"AD-030","name":"tau-bench","family":"evaluation","kind":"Benchmark paper","mechanism":"Evaluates repeated success in stateful, policy-bearing interactions among a user simulator, agent and tools.","evidence":"The paper distinguishes succeeding on every repeated attempt (pass^k) from finding at least one successful attempt.","limitations":"Final-state equality can miss process violations and simulated-user errors can affect results. Historical scores are not present capability estimates.","textdata_use":"Measure reliability across repeated research tasks while also inspecting process and evidence quality.","textdata_evaluation":"Does an arrangement repeatedly retain the same study commitments without hiding substantively different errors?","credit":"Yao et al. (2024).","source_version":"v1 June2024","primary_locator":"arxiv.org/html/2406.12045v1#S5","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"v1 June2024","locator":null,"observed_at":null,"source":"arxiv.org/html/2406.12045v1"}],"primary_source":"arxiv.org/html/2406.12045v1","repositories":[]},{"id":"AD-031","name":"GEPA","family":"evaluation","kind":"Paper and repository","mechanism":"Uses execution feedback to reflect on and evolve prompts while selecting among candidate trade-offs.","evidence":"The conference paper keeps task-model weights fixed and evaluates prompt evolution across several benchmarks.","limitations":"Advantages vary by task; optimization exposure and evaluation selection matter. Bounded transfer evidence does not imply future-model invariance.","textdata_use":"Explore revisions to reusable prompts while keeping evaluation materials and prior variants traceable.","textdata_evaluation":"Does an optimized prompt improve held-out source fidelity without sacrificing another required criterion?","credit":"Agrawal et al. (2026 conference version) and GEPA contributors.","source_version":"conference version of record","primary_locator":"proceedings.iclr.cc/paper_files/paper/2026/file/0e9e708b6f48e14fd0ac29e167413f76-Paper-Conference.pdf#page=6","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"conference version of record","locator":null,"observed_at":null,"source":"proceedings.iclr.cc/paper_files/paper/2026/file/0e9e708b6f48e14fd0ac29e167413f76-Paper-Conference.pdf"},{"version":"v0.1.4 / short commit 8b0ce6c","locator":"GitHub: gepa-ai/gepa/releases/tag/v0.1.4","observed_at":null,"source":"GitHub: gepa-ai/gepa/releases/tag/v0.1.4"},{"version":"tag-scoped MIT text","locator":"GitHub: gepa-ai/gepa/blob/v0.1.4/LICENSE","observed_at":null,"source":"GitHub: gepa-ai/gepa/blob/v0.1.4/LICENSE"},{"version":"open when inspected","locator":null,"observed_at":null,"source":"GitHub: gepa-ai/gepa/issues/379"}],"primary_source":"proceedings.iclr.cc/paper_files/paper/2026/file/0e9e708b6f48e14fd0ac29e167413f76-Paper-Conference.pdf","repositories":["GitHub: gepa-ai/gepa"]},{"id":"AD-032","name":"MIPRO and MIPROv2","family":"evaluation","kind":"Paper and implementation","mechanism":"Optimizes module-level instructions and demonstrations against downstream task scores.","evidence":"The paper studies multi-stage language-model programs; current MIPROv2 documentation is a distinct software manifestation.","limitations":"Results depend on the program, metric and data split; two small paper tasks lack a separate development split. Paper and current API are not interchangeable evidence.","textdata_use":"Keep prompt optimization data separate from held-out assessment of a measurement procedure.","textdata_evaluation":"Compare optimized and manually specified procedures on untouched organizational texts and inspect retained examples.","credit":"Opsahl-Ong et al. (2024) and DSPy contributors.","source_version":"v2; EMNLP 2024 relation in arXiv metadata","primary_locator":"arxiv.org/html/2406.11695v2#S4.SS3","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"v2; EMNLP 2024 relation in arXiv metadata","locator":null,"observed_at":null,"source":"arxiv.org/html/2406.11695v2"},{"version":"mutable API documentation","locator":null,"observed_at":null,"source":"dspy.ai/api/optimizers/MIPROv2/"},{"version":"DSPy3.1.3; closed","locator":null,"observed_at":null,"source":"GitHub: stanfordnlp/dspy/issues/9308"},{"version":"DSPy3.0.3; closed","locator":null,"observed_at":null,"source":"GitHub: stanfordnlp/dspy/issues/9039"},{"version":"EMNLP 2024 VOR, pp. 9340–9366","locator":"aclanthology.org/2024.emnlp-main.525/ ; Title/authors, venue, date, pages and DOI","observed_at":"2026-09-15T01:34:18+00:00","source":"aclanthology.org/2024.emnlp-main.525/"},{"version":"EMNLP 2024 VOR, pp. 9340–9366","locator":"aclanthology.org/2024.emnlp-main.525.pdf ; Appendix B.1 and Table 3, PDF pp.13–14 (printed pp.9352–9353)","observed_at":"2026-09-15T01:34:18+00:00","source":"aclanthology.org/2024.emnlp-main.525.pdf"}],"primary_source":"arxiv.org/html/2406.11695v2","repositories":["GitHub: stanfordnlp/dspy"]},{"id":"AD-033","name":"AI Agents That Matter","family":"evaluation","kind":"Evaluation paper","mechanism":"Frames agent assessment jointly in terms of task accuracy, cost and intended generality.","evidence":"The paper distinguishes development expenditure from per-use inference and discusses holdouts appropriate to intended use.","limitations":"These evaluation principles do not certify that a particular benchmark captures organizational research quality.","textdata_use":"Assess scholarly output alongside model calls, elapsed time and researcher effort.","textdata_evaluation":"Does a more elaborate arrangement improve the intended research outcome under a meaningful resource comparison?","credit":"Kapoor et al. (2024); inspected arXiv v1.","source_version":"v1 July2024; later manifestations not used for claims","primary_locator":"arxiv.org/html/2407.01502v1 ; §3 fixed/variablecost;§4 downstream evaluation;§5 holdoutgenerality","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"v1 July2024; later manifestations not used for claims","locator":"arxiv.org/html/2407.01502v1 ; §3 fixed/variablecost;§4 downstream evaluation;§5 holdoutgenerality","observed_at":null,"source":"arxiv.org/html/2407.01502v1"}],"primary_source":"arxiv.org/html/2407.01502v1","repositories":[]},{"id":"AD-034","name":"Handoff Debt","family":"coordination","kind":"Empirical paper","mechanism":"Compares repository-only, raw-trace, summary and structured-note views at frozen worker-takeover points.","evidence":"The v2 study uses software tasks and successor models to test how inherited context affects continuation.","limitations":"Resolution gains vary; note-generation costs and execution context qualify the comparison. No note format is uniformly best, and the task is software continuation.","textdata_use":"Pass the question, source references, completed work and unresolved interpretations to a successor researcher or agent.","textdata_evaluation":"Can a successor recover both the result and the reason a competing interpretation remained open?","credit":"Handoff Debt authors as identified in the linked primary paper.","source_version":"arXiv 2606.02875 v2, 2026-08-30.","primary_locator":"arxiv.org/html/2606.02875 §3–6 Tables2–4; Limitations","repository_status":"Link for inspection; no installation, license clearance or latest-release claim","evidence_basis":"Documented source summary","reading_scope":"The entry draws on a documented source summary. A full-text review is not established for this entry.","implementation_status":"Not executed or benchmarked for this database.","references":[{"version":"v2 2026-08-30","locator":"arxiv.org/html/2606.02875 §3–6 Tables2–4; Limitations","observed_at":null,"source":"arxiv.org/html/2606.02875"}],"primary_source":"arxiv.org/html/2606.02875v2","repositories":[]}]}