close fullscreen

Knowledge Representations+MORK (MeTTa Optimized Reduction Kernel)+MORK Deep Dive

help edit space_dashboard
Approved by Ursula Addison on 2026-06-15
ai generated
more_horiz
open_in_full page
fullscreen modal
edit edit
space_dashboard advanced
ai reviewed
more_horiz
open_in_full page
fullscreen modal
edit edit
space_dashboard advanced
trie-indexing
more_horiz
open_in_full page
fullscreen modal
edit create
space_dashboard advanced
Authors: Ben Goertzel
Contributors: Adam Vandervorst, Remy Clarke, Luke Peterson, Igor Malovitsa, Marcin Rzeźnicki, Surafel Fikru

New to MORK? Start with the MORK index card for an overview and the MORK Primer for a plain-language on-ramp — this page is the expert-level technical deep dive.

MORK (MeTTa Optimized Reduction Kernel) is Hyperon's high-performance hypergraph engine — a specialized in-RAM processing kernel built for large speedups over previous AtomSpace implementations. It provides the computational substrate on which PRIMUS's cognitive algorithms execute at scale.

On this page

  • Core Mechanisms — the PathMap triemap, the Zipper Abstract Machine, and MM2 Gather-Process-Scatter execution
  • Formal Foundations and Indexing — the Selectivity Theorem and slot-centric indexing
  • Architecture and Ecosystem — the 9-member workspace, the sibling repos, and how the cognitive algorithms plug in
  • Status and Resources — what compiles and runs today, what is under development, and primary sources

Core Mechanisms

PathMap (Triemap Storage)

MORK's foundational data structure is a prefix-compressed triemap (radix tree) that stores S-expressions as paths. Where a traditional graph database scatters nodes and links across memory, MORK organizes them into a structured hierarchy where shared prefixes are stored once. This provides:

  • Content addressing: Each atom is identified by its path in the trie (hash-consing), yielding structural sharing and \(O(1)\) pointer equality
  • Near-constant-time neighbor lookups: PathMap supplies a path algebra (prefix scans, union/intersection of posting lists, anti-join) that directly supports join planning and streaming candidate generation
  • Automatic deduplication: Identical subexpressions share storage automatically
  • Slot-centric views: For \(k\)-ary relations, MORK maintains at most \(k\) slot-anchored prefix views rather than \(k!\) permutations

Zipper Abstract Machine (ZAM)

The execution model that exploits PathMap: given a query with one or more legs (rows of guarded reads), ZAM chooses selective path prefixes, fetches posting lists from the trie, computes intersections incrementally, and streams only the compatible frontier to the unifier. This pushes most pruning before unification. ZAM uses zippers (cursor-based navigation) for efficient, parallel logical inference with near-linear performance scaling across cores.

MM2 (Minimal MeTTa 2)

A low-level dataflow language for defining performance-critical components directly on MORK's data structures. MM2 uses a "Gather-Process-Scatter" paradigm with priority-based execution, sources (inputs), and sinks (outputs). The language is formally decomposed into four semantic segments: a monotonic base (metagraph rewrites where automatic pattern reordering is safe — enabling 1000× speedups without changing semantics), sinks (non-monotonic output operations), sources (input/data loading), and syncs (conjunctions of sources and sinks). Operations include fork/join patterns, set operations, and macro-based partial evaluation. MM2 is where PLN factor-graph message passing, ECAN attention sweeps, pattern mining iterations, and tensor operations run at database-engine speeds.

MORKL

A declarative query language providing "bare metal" access to MORK's trie structures for structural manipulation using S-expression syntax. Think of it as SQL for MORK — whereas MM2 is for dynamic algorithms at scale.

Formal Foundations and Indexing

The Selectivity Theorem

The "MORK theory" paper provides a formal analysis of when ZAM/MORK yields substantial advantage. The selectivity exponent:

\[\gamma(p) = -\log_N\!\left(\frac{|P(p)|}{N}\right)\]
  • Variables: \(p\) = a prefix in the trie, \(|P(p)|\) = size of the posting list for prefix \(p\), \(N\) = total number of atoms
  • Meaning: Measures the normalized information content of a prefix — how much it narrows the search space
  • Source: Goertzel (2025), Articulating Conditions Where ZAM/MORK Yield Benefit

For a \(k\)-leg join over prefixes \(p_1, \ldots, p_k\), under \(\varepsilon\)-independence assumptions:

  • If \(\sum \gamma(p_i) > 1\): the selectivity analysis predicts \(O(1)\) expected candidate intersections — ZAM feeds a constant number of candidates to the unifier
  • If \(\sum \gamma(p_i) < 1\): the expected intersection grows as \(N^{1-\sum\gamma}\), still sublinear

Under a hierarchical generative model (tree-structured conditionals with exponential decay of prefix probabilities), even moderately deep bindings across a few legs push the selectivity sum past 1. Real-world AtomSpaces — logic proofs, program ASTs, semantic parses, knowledge graphs — are approximately hierarchical: symbols follow heavy-tailed distributions, structures are compositionally generated, and query legs bind weakly dependent positions.

Slot-Centric Indexing and the Break-Even Rule

Instead of \(k!\) permutations, MORK maintains at most \(k\) slot-centric prefix views — one per argument position — plus selectively promoted "hot" pair views. Path-key format examples (binary relation):

Canonical:  R/_1/<EXPR_ID>/_2/<C_ID>  → payload
Flip view:  R/_2/<C_ID>/_1/<EXPR_ID>  → pointer to canonical

The probabilistic break-even rule:

\[p_{s+1}(M - L) > \alpha\]
  • Variables: \(p_{s+1}\) = probability a query anchors on the \((s{+}1)\)-th slot; \(M\) = mining cost when anchor is missing; \(L\) = direct prefix-view hit cost; \(\alpha\) = storage cost of one additional view
  • Meaning: Materialize the next slot view only when expected cost reduction exceeds storage cost

Worked example: with \(p_1=0.4, p_2=0.6, L=1, M=50\), the flip view yields a 29.4× expected speedup for roughly 2× index entries. For hot multi-slot queries, the general problem is submodular coverage under a storage budget, solvable greedily. (Source: Goertzel 2025, MORK Slots)

Architecture and Ecosystem

Crate Structure

MORK is a 9-member Rust workspace (MORK/Cargo.toml; requires a nightly Rust toolchain):

  • interning/ — Symbol interning with lock-free handles (128 concurrent writers)
  • expr/ — S-expression types, binary encoding, macros
  • frontend/ — Multiple parsers (CZ2, CZ3, HE, Rosetta, bytestring formats)
  • kernel/ — Core Space implementation, sinks, sources, pure reduction engine
  • experiments/eval/ — Exploratory MM2 evaluator
  • experiments/eval-ffi/ — FFI-side evaluator integration
  • experiments/eval-examples/ — Example workloads
  • experiments/unification_test_laws/ — MORK ↔ SWI-Prolog unification correctness audit (PR #49 Prolog-as-oracle for the unification subset only)
  • linalg/ — Sparse and dense linear algebra: CSR SpGEMM, block-sparse SIMD attention, and a runtime einsum VM (see Linear Algebra / Tensor Logic section below)

Foundational dependency: PathMap — sibling repo authored by Luke Peterson; declared as ../PathMap/ with jemalloc, arena_compact, nightly features. PathMap is the low-level trie substrate (key-value store with prefix compression, structural sharing, algebraic operations); MORK's path-algebra and zipper machinery sit on top of it. See PathMap for substrate details.

Server branch: The mork-server deployment is maintained on a separate server branch, distinct from main. DAS pins MORK to an older commit via das/src/docker/mork/Dockerfile.server; the live server branch has diverged with deadlock and UTF-8 fixes applied. See Status for drift detail.

System Interfaces

  • PeTTa: Primary MeTTa compiler connecting via FFI. PeTTa alone handles 50–100M atoms; PeTTa/MORK has been benchmarked up to 400M atoms in RAM (documented successful 100M/200M/300M/400M loads; 500M ran out of memory in the same benchmark). Earlier wiki text quoting "500M+ atoms in RAM" treated the OOM ceiling as demonstrated capacity — corrected.
  • mork_ffi: Rust FFI bindings for SWI-Prolog. 65s for 1M atoms (vs OOM with predicate store). Two distinct Prolog roles: (1) PR #49 unification oracle in experiments/unification_test_laws/; (2) PeTTa runtime bridge via SWI-Prolog predicate mork/3 (see mork_ffi/mork.c and morkspaces.pl).
  • faiss_ffi: FAISS vector similarity FFI for Prolog, enabling structural random indexing.
  • MeTTa-IL: Compilation target — routes execution to MORK for local low-latency reasoning.
  • DAS: DAS contains a code-real MorkDB AtomDB backend (subclass of RedisMongoDB) that talks to a MORK HTTP server (server-branch deployment) and Mongo-side metadata. Link/S-expression delete is hard-failed in the MorkDB implementation ("MORKDB does not support deleting links") — DAS-as-MORK-backend is integration-ready for loads and queries, NOT a Decko-compatible mutable store. See DAS Full.
  • Linear Algebra / Tensor Logic (linalg crate): Merged into MORK in 2026-06 (the linalg/ crate in trueagi-io/MORK).einsum An Einstein-notation () kernel expresses sparse (CSR) and dense tensors in one expression with dynamic per-dimension sparse↔dense switching at a density cutoff (~1.5%); a JIT compiler emits native machine code (AVX), reported ~290× over the interpreted prototype and on par with hand-written kernels (CPU≈GPU price/performance for matmul, currently F32). Applied e.g. to transitive closure via repeated graph powers. (MORKification Weekly 2026-04-27 → 06-15.)
  • IO / Resource interface (supersedes ByteFlow and ShardZipper): The earlier ByteFlow (adaptive block packing) and ShardZipper (Merkle-based distributed state) experiments were retired — ByteFlow judged infeasible without rebuilding MORK, and ShardZipper's GPU-offload mechanics lifted from the PathMap level into a single narrow IO resource interface at the MORK level (the same resource API now backs CPU multithreading, GPU/float acceleration, strings, external programs, network, and a live Z3 theorem-prover sink). (MORKification Weekly 2026-04-27 / 05-04; neither term remains in the MORK codebase as of last verification.) Historical detail: RAPTL had enhanced partitioning with a triple quantale \((\varphi, \alpha, r)\) and confidence-weighted scoring: \(\text{partition\_score}(s) = \text{avg\_confidence}(s) / (\text{predicted\_cost}(s) \cdot \text{locality}(s))\). (Goertzel 2025, RAPTL ShardZipper §3.2–3.3)

MORK Special Forms (MeTTaTron)

MeTTaTron provides four special forms that bridge high-level MeTTa and low-level MORK execution. All use uniform conjunction semantics — the (,) wrapper makes result cardinality explicit and enables meta-programming:

  • exec (<priority> <antecedent> <consequent>) — Rule execution with conjunction antecedents. All antecedent goals must match (left-to-right, variable bindings threaded through). Consequents can be conjunction results or space-modifying Operations (O (+ fact) (- fact)). Priority determines execution order. Non-deterministic: multiple antecedent solutions produce multiple consequent evaluations.
  • coalg (<pattern> <templates>) — Coalgebra patterns for tree transformations. Template conjunction cardinality determines result count: (,) = zero results (termination), (, t) = one result, (, t1 t2) = unfold to two. Enables hierarchical decomposition (e.g., tree → contexts → leaf values via lift/explode/drop stages).
  • lookup (<pattern> <success-goals> <failure-goals>) — Conditional fact queries with branching. Variables bound during pattern match are available in the success branch. Nestable for priority chains.
  • rulify ($name <pattern> <templates> <antecedent> <consequent>) — Meta-programming: generates exec rules from coalgebra definitions by pattern matching on template arity. Enables runtime rule generation from declarative specifications.

The conjunction pattern provides ~36% parser code reduction, ~40% evaluator simplification, and ~80% fewer edge-case bugs, with negligible runtime overhead (~2 bytes per conjunction, ~10ns per goal evaluation).

(Provenance: repo-doc, MeTTa-Compiler MORK special forms documentation)

Key Cognitive Algorithm Integrations

  • PLN: Backward chaining (HeadIndex/FactIndex/UnifyIndex) and factor-graph belief propagation (FactorAtom/VariableAtom with near-constant-time neighbor lookups) — paper/proposal/benchmark-only at this snapshot. 2026 source-code reviews of PLN and of the MORK backend both confirm no code-real FactorGraph PLN over MORK in the inspected primary repos (MORK/, PathMap/, mork_ffi/, mork-rust-sdk/, mork-ts-sdk/: 0 hits for FactorGraph/factor_graph/belief_propagation/pln/wmpln/lib_pln/AttentionBank/cog-av-sti).
  • WILLIAM: Trie nodes carry occurrence counts, subtree totals, compression-gain sums for real-time pattern detection
  • Weighted Atom Sweeps (implementation analogy, NOT ECAN currency): The weighted-atom-sweep repo is a separate adjacent experimental crate outside the canonical MORK 9-member workspace, depending on PathMap and MORK expr. Its AtomHeader is a generic trait — NOT an STI/LTI/TruthValue/AttentionValue — and its match counter is an integer, not an ECAN attention currency. The pattern (aggregate weight counters in trie nodes for importance-proportional sampling) is repurposable for recency/priority/card-version metadata if a Decko-specific AtomHeader and traversal policy are defined, but NOT a code-real ECAN bridge at this snapshot.
  • MOSES/GEO-EVO: Program templates as content-addressed atoms; near-identical candidates deduplicated automatically
  • Pattern Mining: Patterns as subtree traversals; counts as capsule summaries at nodes

Implementation Anchors

  • MORK (primary kernel) — 9-member Rust workspace; nightly toolchain required.
  • PathMap (foundational substrate, sibling repo) — Luke Peterson's prefix-compressed triemap; declared as MORK ../PathMap/ dep. See PathMap.
  • MORK server branch (deployment line) — mork-server at origin/server; DAS pin and image-tag reconciliation discussed at Status.
  • CZ2 — Scala 3 triemap toolkit and scaling experiments.
  • mork_ffi — Rust FFI bindings for SWI-Prolog/MeTTa integration.
  • weighted-atom-sweep (adjacent experimental crate, NOT in canonical MORK workspace) — Rust weighted atom sweep on PathMap; implementation analogy for ECAN-style sampling but no ECAN/AtomSpace Value bridge code at this snapshot.
  • MM2_Structuring_Code — Comprehensive MM2 tutorial (28 examples).

Status and Resources

Last verified: 2026-06-01

Current Status

  • Operational: Triemap storage, ZAM execution, bidirectional pattern matching, MM2 language, PeTTa/MORK integration, mork_ffi for Prolog bridging
  • Under development: MORK-native PLN (backward chaining + factor graphs — paper/proposal/benchmark-only; a 2026 review of the MORK Rust source found no code-real FactorGraph PLN at this snapshot), WILLIAM trie instrumentation, and the IO/resource interface that retired the earlier ByteFlow GPU-offload and ShardZipper distributed-state experiments.
  • Recently shipped (2026-06; previously listed here as "under development"): the linalg tensor-logic crate (Einstein-notation sparse/dense kernels with an AVX JIT, ~290× over the interpreter) and a new asymptotically-optimal query-compilation path — MM2 queries lower to a fixed-memory, order-independent state machine that visits each trie location at most once, exposing a first-class stream set-algebra operator (intersection / union / subtraction; the operator itself was still landing as of mid-2026). This supersedes the earlier "streaming fusion (~10× slower)" status. (MORKification Weekly 2026-05-11 → 06-15.)
  • Proposed: Multi-machine distributed processing, QuantiMORK (neural tensor encoding), WASM edge deployment, native MeTTa-to-machine-code compiler

Implementation Findings (transcript-backed, MORKification Weekly Aug 2025–Apr 2026)

  • MM2 scale: ~350 grounded functions as of Jan 2026. One MM2 step triggers billions of parallel rewrites due to massive parallelization.
  • Sources/sinks architecture: Three-layer resource abstraction — resources, sources (readers), sinks (writers). This is the integration surface for ECAN weights, hypervectors, and external systems. Note: CountSink in the kernel is an MM2 query/reduction primitive (per-execution accumulator), not a persistent revision log — do not target it as a Decko card-history counter.
  • Compression benchmarks: PathMap uses seven levels of nested shared patterns to represent all 64-bit integers in 8 nodes. JSON import: 20× reduction (780 GB JSON → 40 GB ACT).
  • RAM scaling benchmark: PeTTa/MORK has been demonstrated up to 400M atoms in RAM (documented successful 100M/200M/300M/400M loads; 500M ran out of memory in the same benchmark). Earlier "500M+ atoms in RAM" wiki text treated the OOM ceiling as demonstrated capacity — corrected.
  • Streaming fusion investigation: Six implementations tested — all ~10× slower than binary operations due to branch traversal overhead. Active investigation with database-inspired query optimization.
  • Concurrency advantage: MORK surpasses ATRIUM (20,000 threads) on its own benchmarks due to sequential thread coordination rather than threads fighting over shared memory.
  • Applications: 4×4 Sudoku (~2 ms), CTL model checking, decision tree learning, Blocks World / PDDL planning.

Known Limitations (discussion-backed, MORK Mattermost)

  • Concurrency ceiling: MORK crashes at ~200 concurrent users (Rejuve.Bio load test, Mar 2026) rather than degrading gracefully.
  • No automatic persistence: Data lost on restart. Manual save/restore via paths_export() / paths_import().
  • Negative querying unsound: Removed from MM2. Use != or nested if/not/find instead.
  • Memory multiplier: ~64 bytes/atom. String-heavy datasets need interning (7 GB → 1.9 GB vs 31 GB default).
  • WASM deprecated: ~15× overhead. Pure Rust grounded functions now default.
  • Server-branch versioning: The mork-server deployment line is maintained on a separate server branch. As of last verification, three references were not reconciled: das-toolbox CLI defaults to image tags trueagi/das:mork-server-1.0.5 + mork-loader-1.0.5; das/src/docker/mork/Dockerfile.server pins MORK to a 2025-07 commit; and the live server branch has diverged substantially ahead of the DAS pin. Production deployment must reconcile all three references. Notable post-pin fixes in the gap include: server shutdown deadlock, user-status-map cleanup, a lock-held-too-long deadlock, UTF-8 validation for the symbol pathway, and edge-case/malformed-symbol test coverage.
  • Link/S-expression delete unsupported in DAS-MorkDB backend: The MorkDB implementation hard-fails on link delete. Node delete works (inherited from RedisMongoDB); link delete does not. flush_pattern + re_index_patterns provide batch-rebuild workarounds, NOT live mutable-store CRUD. See DAS Full.

Open Problems / Research Directions

  • Multi-machine distribution — scaling across clusters while preserving PathMap locality
  • QuantiMORK — wavelet/multiresolution DAG encoding for neural structures
  • GPU/TPU acceleration — dense/sparse numerical kernels shipped on CPU as the linalg crate (AVX JIT) behind the IO/resource interface (ByteFlow itself was retired); a GPU back end on the same interface remains open
  • Unifier correctness — the Vampire-derived unifier is now validated against SWI-Prolog over a ~100k-axiom oracle (PR #49), and a long-standing occurs-check / cyclic-term bug (issues #29/#43) was fixed and merged to main; a cascade of further unification edge-case bugs (two unification algorithms, both requiring cycle checks) was still being resolved as of mid-2026 (MORKification Weekly 2026-04-20 → 06-22)
  • Community and third-party package ecosystem
  • Formal verification of ZAM correctness properties
  • Decko-compatible mutable-backend semantics — link delete, transactional history, RichText/file/permission mappings; MORK alone does not provide them, an adapter layer is required

Primary Sources

  • Goertzel, B. (2025). Hyperon for AGI⇒ASI Whitepaper, §2.3, §3.6.
  • Peyton Jones, S. et al. Triemaps that Match.
  • Internal working papers: the July–October 2025 MORK research working-papers (ZAM/MORK benefit conditions, path algebra to tensor logic, slot-centric indexing) are indexed with provenance and status qualifications in MORK Research Family Publication Map.
  • Theory sources: the selectivity theorem and related MORK theory sources are mapped in MORK Theory Publication Map.
  • Engineering provenance: the engineering corrections on this card (9-member workspace, 400M RAM ceiling, server-branch reconciliation, MorkDB delete limitation) derive from a source-code review of the AtomSpace backend integration conducted 2026-04-29.

Source map: See MORK Research Family and MORK theory publication map for the working-paper family and theory sources, including the Hyperon Whitepaper, selectivity theorem, path-algebra/tensor-logic, slot-centric indexing, and Triemaps background.

Status: Current. MORK is operational — the 9-member Rust workspace (requires a nightly Rust toolchain), with working triemap storage, Zipper Abstract Machine, bidirectional pattern matching, and MM2 execution. PeTTa/MORK integration handles 400M+ atoms in RAM (400M is the reconciled figure; an earlier 500M figure on the wiki was inaccurate). MORK-native PLN execution remains paper/proposal/benchmark-only in the current review; WILLIAM resource-interface integration and distributed multi-machine execution are open research directions not found as integrated runtime features in this review. Note that PathMap is a sibling-repo dependency (separate codebase), not a MORK subcrate; the MORK server-branch is versioned independently of the main library.

Where MORK sits. In the Hyperon AtomSpace family, MORK is the high-performance Layer-4 backend — the four layers run from the abstract AtomSpace concept, to the in-memory Hyperon Space, to the distributed DAS, to MORK. The Architecture and Ecosystem and Status and Resources subcards cover the engineering detail: the 9-member workspace, the independently-versioned server branch, the 400M-atom RAM scaling, the MorkDB link-delete limitation that currently blocks promoting MORK to a primary store, and why MORK's weighted-atom-sweep work is best read as an experimental analogy to ECAN rather than a literal port of it.

Related cards: AtomSpace (abstract concept), DAS (distributed complement), PathMap (foundational trie substrate, sibling-repo dependency), PLN (chaining + factor graphs on MORK), ECAN (weighted atom sweeps), MORK Theory Publication Map

Discussion