Ingestion Agent Handoff

help edit space_dashboard

Ingestion Agent Handoff — Publications Backlog

Handoff from the publications-coverage-audit / synthesis sessions (orchestration: Claude + Codex + Gemini) to the bulk-ingestion pipeline agent. Snapshot 2026-06-09. This card defines the division of labor + the two concrete asks the audit explicitly punted to the pipeline. The line-level backlog lives in the sibling tracker Sources Pending Analysis (8160); the pipeline’s own build contract is Spec 17646 + Impl Plan 17648 (Magi Archive).

Division of labor (the contract)

Layer Owner Rule
Bulk RawData ingestion Pipeline (you) Pull direct from source platforms (Mattermost / Otter / Fireflies / mailing lists / sheet), store complete + unfiltered raw text, server-side direct-to-Decko-DB — not via MCP upload, not from manually-created intermediate cards.
Synthesis layer (Publications+ / Publication Maps+ family cards) Synthesis sessions Family-level synthesis, cross-linking, expert-bar curation. Template = Publication Maps+MORK Research Family (8441). The pipeline must NOT create synthesis cards.
Filtering / classification Separate downstream agents Don’t filter at ingest — store everything raw; classification is a later, dedicated step.

RawData is not 1-to-1 with Published. RawData is the substrate; the synthesized cards are the product. Stocking RawData is the pipeline’s job; turning families of RawData into navigable knowledge is the synthesis sessions’ job.

State handed over (from the 2026-06-09 coverage audit)

  • 214 RawData+Publications cards already exist + 56 Publications paper cards. This is a reconcile + enrich state, not ingest-from-zero.
  • arxiv = FALSE GAP: all ~30 arxiv_*.pdf are already carded. The “ID-named PDF” flag was a filename↔title matching artifact. Lesson: verify title↔filename matches before creating — the heuristic “missing” counts overstate.
  • 144 canonical-sheet dated rows still lack a card (55 curated 2019-2025 + 89 scholar 1990-2000 back-catalog) — candidate bulk-ingest targets, but mostly co-authored, non-Goertzel-core conference/workshop papers (lower priority; P3 in tracker 8160).
  • ~145 of 154 publication_texts/mattermost_papers/ are un-synthesized RawData — these are tight multi-version research families (MORK, MeTTa specs, SubRep, Omegaclaw PLN, …), handled at the synthesis layer as families, not one-by-one.

Two concrete asks for the pipeline

  1. Year / venue metadata schema (PRIMARY). The 2001-2018 OpenCog-era is substantially covered by the wiki Publications tree, but years are inferred, not stored as machine-readable card metadata — so the era isn’t queryable. Add a year + venue field to publication_texts/*.json and to the Publications/RawData cards. This makes the whole bibliography sortable/filterable by era and closes the audit’s main structural gap.
  2. Bulk-card the 144 missing sheet rows (direct-to-DB), ideally the curated 2019-2025 subset (55) first since it’s fully curated (tags/venue/DOI/abstract). Verify title↔filename before each create to avoid duplicate cards (arxiv false-gap lesson).

Coordination / collision rules

  • One-writer-only per card. The pipeline owns the bulk RawData layer; synthesis sessions own the synthesis layer. Don’t write synthesis cards; we won’t bulk-card raws.
  • Git: the shared working tree is on ui/draft-published-tag-pills (your branch) with uncommitted files. Orchestration sessions commit to main via isolated git worktree and will not branch-switch your tree. Keep your pipeline work on your branch.
  • Tracker 8160 (Sources Pending Analysis) is the authoritative publications backlog dashboard — work from it; update it as buckets clear.
  • NOT your job: the Cross-Org Cognitive Repo Sweeps program (tracker 8487) is a source-code extraction program (GitHub/GitLab org sweeps), entirely separate from publications ingestion. Ignore it.

Source artifacts

  • Backlog dashboard: tracker 8160 (this parent).
  • Archive snapshots (local repo, dated): scripts/archive/publications_coverage_audit_2026_06_09.txt, …/sources_pending_analysis_2026_06_09.txt, …/arxiv_reconcile_2026_06_09.txt.
  • Pipeline build contract: Spec 17646 (v0.5) + Impl Plan 17648 (v0.4) on Magi Archive; prod state per memory project_ingestion_pipeline_prod_state_2026_06_04.