DAS Full

help edit space_dashboard
Draft — This content has not been approved for publication.

← Back to DAS

Responsible: André Senna

Papers: Goertzel (2025), Hyperon Whitepaper §2.5–2.6; Goertzel (2025), TrueAGI AI State Management Platform: Initial Design

Status: Current. The DAS core (query engine, AtomDB, attention broker) is operational with Redis+MongoDB and in-memory backends. The MeTTa parser and CLI toolbox are functional. MORK integration is code-real with caveats (see Storage Backends and the AtomSpace cluster-pilot wording immediately below). The broader State Management Platform with CRDT geo-replication, Rholang lens contracts, and Kubernetes orchestration is proposed.

This card provides technical depth beyond the concise DAS index card. DAS (Distributed AtomSpace) is Hyperon's distributed knowledge management system — a high-speed, dynamic memory fabric for massive, mutable hypergraphs. It decouples persistence from attention dynamics, enabling AtomSpaces to span multiple machines while maintaining the cognitive properties needed for AGI.

AtomSpace Cluster-Pilot Lock-In (2026-04-29)

DAS sits at Layer 3 of the four-layer AtomSpace taxonomy locked in by the AtomSpace Backend Integration Cluster Pilot (see AtomSpace Full → Implementations). The cluster pilot's R4.L1 finding is the canonical technical wording for DAS's MORK coupling and AttentionBroker character:

DAS contains a code-real MorkDB AtomDB backend, but it is a SPLIT IMPLEMENTATION: DAS-side AtomDB code is real (singnet/das: MorkDB.h, MorkDB.cc, AtomDBSingleton.cc); the MORK server is pinned through a Docker/server-branch path; link/S-expression delete is not supported in the MorkDB backend (MorkDB.cc:268-270 hard-fails with "MORKDB does not support deleting links"). Treat MORK-as-DAS-backend as experimental/integration-ready for loads and queries, not a Decko-compatible mutable store.

AttentionBroker stores per-context STI outside AtomDB, updates exact-count Hebbian networks, and uses a token rent/wages/spreading model; it is an ECAN-compatible engineering surrogate, NOT the literal 2009 ECAN formula set. Implementation: ExactCountHebbianUpdater (HebbianNetworkUpdater.cc:57-96, weights = count(A→B)/count(A)) + TokenSpreader fixed-token rent/wage/spreading model (StimulusSpreader.cc:54-60 rent = rent_rate * importance; :140-157 wages proportional to request counts; :75-82 deterministic arity-weighted spreading; fixed defaults RENT_RATE=0.75, lower/upper spreading=0.10).

Hyperon Experimental bridge: new-das! MeTTa op constructs a DistributedAtomSpace and returns a DynSpace (hyperon-experimental/lib/src/metta/runner/builtin_mods/das.rs:156-199). The das feature is default-enabled in lib/Cargo.toml:45-55, depending on metta-bus-client from singnet/das tag 1.0.2.

Server-branch drift caveat (2026-04-29): DAS Dockerfile pins MORK commit 578a759 (2025-07-21); local origin/server is at 5b04a1d (2026-04-18) with 49 subsequent commits including server-shutdown deadlock, lock-held-too-long deadlock, UTF-8 validation, and edge-case malformed-symbol fixes. das-toolbox CLI defaults to image tags trueagi/das:mork-server-1.0.5 and mork-loader-1.0.5 — three potentially-different references (image tag, Dockerfile pin, server-branch HEAD) that production deployment must reconcile.

The DAS-AttentionBroker classification places it in the same lineage as Classic OpenCog ECAN C++ and the metta-attention MeTTa port (0/4 strict-literal compliance against the 2009 ECAN paper across all three; see ECAN cluster pilot finding #1 in the top-level CLAUDE.md). DAS is more production-oriented but is the same engineering-surrogate family.

Core Mechanisms

Separation of Concerns. DAS's key architectural insight is decoupling two timescales of knowledge management:

  • Long-Term Importance (LTI): Vast persistence stored in distributed backends (Redis+MongoDB clusters, RocksDB, or MORK). This is the system's durable memory — everything it has learned and experienced.
  • Short-Term Importance (STI): Immediate dynamics managed in high-speed RAM by the Attention Broker. This is the system's working memory — the subset of knowledge currently relevant to active cognitive processes.

Attention Broker. The Attention Broker mitigates combinatorial explosions inherent in graph traversal. It maintains separate Hebbian networks for different query contexts, each with independent STI values per atom — the same atom can have different importance in different contexts. Hebbian link weights encode co-occurrence probabilities: the probability of atom B appearing in a query answer given that atom A is present. STI updates can be implicit (automatically triggered during query execution as answers flow through the tree) or explicit (user-requested stimulation and activation spreading). This enables the Attention Broker to learn query-relevant importance patterns from the actual queries being processed. (See cluster-pilot lock-in above for the engineering-surrogate vs literal-2009-ECAN distinction.)

Query Architecture. DAS supports multiple simultaneous query agents, each specialized for different cognitive operations: Query Engine (pattern matching), Evolution Agent (fitness-driven search), Inference Agent (PLN-style reasoning), Link Creation Agent (dynamic graph modification), Context Broker (context-sensitive routing), and AtomDB Broker (backend-agnostic storage). A Service Bus mediates inter-component communication in a service-oriented architecture, supporting pluggable transport (GRPC, MQTT, in-RAM queues).

Query Tree Execution. The Query Engine processes queries in three steps: (1) parsing (MeTTa expressions or token vectors), (2) execution planning (constructing an optimized query tree), and (3) query execution (streaming results). Query trees are composed of three element types, each an asynchronous independent processing unit:

  • Sources (leaf nodes) — Fetch candidate answers from AtomDB matching a LinkTemplate pattern, query the Attention Broker for STI values, and emit candidates sorted by decreasing STI
  • Operators (internal nodes) — Apply Boolean logic (AND, OR, NOT) to combine candidate streams from multiple sources, preserving STI ordering
  • Sinks (root) — Deliver final results to the caller, who can iterate or interrupt at any time

Complex queries compose LinkTemplates with Boolean operators in arbitrarily nested expressions (e.g., (or (contains $s (word "aaa")) (contains $s (word "bbb")))). Elements can be deployed as threads, processes on the same server, or across different machines — the communication channel (RAM queues, GRPC, MQTT) adapts to the deployment topology.

Query Evolution. For queries with billions of potential results where only a few optimal answers are needed, DAS provides an evolutionary algorithm inspired by BOA (Bayesian Optimization Algorithm). Rather than mixing individuals with genetic operators, it samples each generation using the Attention Broker's probabilistic criteria: (1) sample a population by executing the query, (2) evaluate a user-provided fitness function on each result, (3) select the fittest individuals to stimulate importance in the Attention Broker's Hebbian network, (4) repeat — each generation biases STI allocation toward atoms that compose high-fitness answers. This leverages DAS's existing attention infrastructure to perform fitness-directed optimization without a separate evolutionary framework.

Deployment Architecture. DAS supports a Lambda Architecture using OpenFaaS (preferred) or AWS Lambda for distributed query execution. A Pattern Inverted Index maps patterns to their knowledge base occurrences, enabling efficient pattern matching without exhaustive graph traversal — analogous to inverted indexes in full-text search but operating over hypergraph patterns rather than text tokens. A Cache Layer sorts and partitions results by relevance, returning iterators that "fetch additional chunks on demand." A Traverse Engine handles hypergraph navigation for remote DAS connections, pre-fetching surrounding atoms to reduce round-trip latency. (Provenance: blog/post, SingularityNET Medium — "The Distributed Atomspace (DAS): A New-age Knowledge Repository", Jun 2024)

(Provenance: repo-doc, DAS conceptual documentation)

Storage Backends

DAS abstracts over multiple storage backends via the AtomDB interface:

  • Redis + MongoDB (current, primary) — Redis for fast key-value lookups and caching; MongoDB for persistent document storage. The standard production deployment.
  • In-memory (current) — For development and testing without external services.
  • MORK / MorkDB (code-real with caveats) — DAS-side MorkDB AtomDB class (subclass of RedisMongoDB at MorkDB.h:43) talks to a MORK HTTP server (deployed from MORK's server branch). Add-side: MorkDB.cc:197-281 generates MeTTa expressions and posts them to MORK while upserting Mongo metadata. Read-side: MorkDB.cc:150-186 converts link schema → MeTTa, calls MORK export, decodes results. Delete-incomplete: MorkDB.cc:268-270 hard-fails on link/S-expression delete; flush_pattern + re_index_patterns provide batch-rebuild only, NOT live mutable-store CRUD. Server pin: das/src/docker/mork/Dockerfile.server pins MORK 578a759 (2025-07-21). See cluster-pilot lock-in above for full server-branch drift caveat.
  • RocksDB (via adapter) — Single-host persistent storage using embedded key-value store.

Expression hashing and HandleTrie data structures provide efficient content-addressed storage and retrieval across all backends.

Mathematical / Formal Foundations

The "TrueAGI State Management Platform" design document describes the mathematical foundations for DAS's next-generation distributed architecture:

  • State-Transition Monoid: Each user interaction or AI update is treated as an endomorphism \(\delta_i : S \to S\), where sequences form a free monoid \(I^*\). A snapshot after inputs \(i_1 \ldots i_n\) is \(s_n = \delta_{i_n} \circ \cdots \circ \delta_{i_1}(s_0)\). This gives a precise way to record, snapshot, and replay state.
  • Coalgebraic Streams: Each AI instance defines streams of outputs and next states as an \(F\)-coalgebra. Final coalgebra uniqueness guarantees consistent replay semantics.
  • CRDT Join-Semilattices: For geo-distributed deployment, the state space \(S\) is equipped with a join-semilattice \((S, \sqcup)\). Concurrent branches merge via \(s_A \sqcup s_B\), ensuring associative, commutative, and idempotent convergence without coordination overhead.
  • Lenses and Permissions: Category-theoretic lenses (get, put) project and update subviews of state, obeying round-trip laws. These map directly to Rholang contracts on MeTTaCycle for permissioned access control.

These foundations are described in the design document as a proposed architecture; the current DAS implementation uses simpler replication and access control mechanisms.

System Interfaces and Dependencies

  • MORK: DAS provides distributed persistence via MorkDB; MORK handles in-memory triemap processing on the server side. See cluster-pilot lock-in above for the split-implementation caveats and server-branch drift.
  • ECAN: The Attention Broker is an ECAN-compatible engineering surrogate (not literal 2009 ECAN); see Attention and Motivation for the broader attention lineage.
  • ASI Chain: In decentralized mode, DAS event streams replicate across ASI Chain nodes with CRDT merging. Rholang contracts enforce permissioned access.
  • MeTTa: DAS is queryable from MeTTa via hyperon-experimental's new-das! op (das.rs:156-199) and the hyperon_das Python package, both routing through the Space API.

Legacy AtomSpace Storage Ecosystem

DAS builds on lessons from the OpenCog-era storage infrastructure (Layer 1 of the AtomSpace taxonomy), which remains available for legacy use:

  • atomspace-storage — Base StorageNode API defining the generic interface for all backends, including ProxyNodes for mirroring, caching, and load-balancing.
  • atomspace-rocks — RocksDB-based persistent local storage (single-user, single-host).
  • atomspace-cog — Network-distributed AtomSpaces via CogServer (TCP/IP, multi-threaded).

DAS represents the Hyperon-native evolution of this infrastructure, designed for larger scale and tighter integration with MORK and the ASI Chain.

Implementation Anchors

  • das (primary) — C++ with Python client (hyperon_das). Bazel build via Docker. Query agents, Attention Broker (ExactCountHebbianUpdater + TokenSpreader), HandleTrie, MorkDB AtomDB backend.
  • das-metta-parser — C Flex/Bison parser loading MeTTa expressions into MongoDB/Redis backends.
  • das-toolbox — Python CLI tools (das-cli) for DAS infrastructure management; defaults to image tags trueagi/das:mork-server-1.0.5 and mork-loader-1.0.5.
  • MORK server branch — Pinned via das/src/docker/mork/Dockerfile.server at MORK commit 578a759; see MORK Full Status for drift detail.
  • Hyperon Experimental DAS feature — hyperon-experimental/lib/src/metta/runner/builtin_mods/das.rs; das feature default-enabled with metta-bus-client from singnet/das tag 1.0.2.
  • Legacy: atomspace-storage, atomspace-rocks, atomspace-cog

Current Status

  • Operational: DAS query engine; AtomDB with Redis+MongoDB AND code-real MorkDB AtomDB backend (with delete-incomplete caveat); Attention Broker (ECAN-compatible engineering surrogate; ExactCountHebbianUpdater + TokenSpreader) with Hebbian activation spreading; MeTTa parser; CLI toolbox; Python client; hyperon-experimental new-das! bridge.
  • Under development: Production reconciliation of MORK server-branch pin / image tags / origin/server HEAD; Decko-compatible mutable-store semantics (link delete blocks live CRUD); improved query agent specialization.
  • Proposed: State Management Platform with CRDT geo-replication, coalgebraic replay semantics, Rholang lens contracts for permissioned access, Kubernetes-orchestrated multi-region deployment.

Open Problems / Research Directions

  • MORK + DAS integration — defining the boundary between hot in-memory processing and cold distributed persistence; reconciling the three server-branch references for production.
  • Live mutable-store CRUD over MorkDB — link/S-expression delete is currently blocking; flush_pattern batch-rebuild is not Decko-compatible.
  • CRDT convergence at scale — ensuring efficient join-semilattice merging for large, rapidly evolving AtomSpaces.
  • Attention Broker scaling — maintaining Hebbian activation efficiency as graph size grows.
  • ASI Chain integration — event stream replication with cryptographic provenance.

Status and Resources

  • Goertzel, B. (2025). Hyperon for AGI⇒ASI Whitepaper, §2.5–2.6: Distributed Execution, State Management.
  • Goertzel, B. (2025). TrueAGI AI State Management Platform: Initial Crude Design and Development Plan.
  • AtomSpace Backend Integration Cluster Pilot (2026-04-29) — cluster archive at scripts/archive/atomspace_pilot/; Source 4 reconciliation R4.L1 is the canonical record for the cluster-pilot wording on this card.

Related cards: AtomSpace Full · MORK Full · ASI Chain Full



Discussion