Architecture overview
This note describes how the Barrel umbrella is split today and where it is headed. Read it when you need to know which application owns a responsibility, or before adding a new application to the umbrella.
Current applications
Barrel is a rebar3 umbrella. Each application is a separate OTP application with a clear responsibility and its own public API.
- barrel_docdb: the document layer. Document metadata, MVCC, the changes feed, replication primitives, attachments, the timeline (branch, PITR, merge), and the catalog/indexing foundation. Usable standalone as an embedded document database.
- barrel_vectordb: the vector layer. The embedded vector database, local ANN indexes (HNSW, FAISS, DiskANN), BM25, and the local hybrid search that combines them. Usable standalone as an embedded vector database.
- barrel_embed: embedding generation across providers (local Python,
Ollama, OpenAI, and others).
barrel_vectordbuses it for text and hybrid search. - barrel_rerank: cross-encoder reranking. Optional, used by
barrel_vectordb. - barrel_faiss: Erlang NIF bindings for FAISS. Optional backend for
barrel_vectordb. Needs the FAISS C++ library, so it is excluded from the default build. - barrel_ngram: trigram substring and regex index over
barrel_docdbdocuments, with immutable checksummed segments.barrel_serverserves it as thengram_searchMCP tool. - barrel_crypto: encryption-at-rest primitives. AES-256-GCM envelope, offset-addressable CTR, HKDF key derivation, and pluggable key providers. Used by docdb and vectordb for encrypted databases.
- barrel: the embeddable database. Composes
barrel_docdb,barrel_vectordb, andbarrel_cryptoso a document, its attachments (blobs), and its vector share one id. Adds record mode, BQL, the database lifecycle manager (barrel_dbs), and contexts (barrel_ctx*: the context catalog, federated queries, working sets, export and read-only import). The timeline itself lives inbarrel_docdb;barrelexposes it on the composed database. Serves no network protocol; its remote context client calls other nodes over HTTP. The hex "edge" package. - barrel_spaces: the agent layer. Spaces (shared context databases),
capability tokens, sessions with TTL, and handoffs, built on
barrel. - barrel_server: the network server. Exposes
barrelandbarrel_spacesover HTTP (REST/JSON) and MCP usinglivery, including the/contextsand/worksetsroutes and thecontext_*MCP tools. Opt-in behind theserverprofile; not part of the default embeddable build. - barrel_att_s3: an S3-compatible attachment backend for
barrel_docdb. Opt-in behind thes3profile.
Intended split
The longer-term shape of the stack separates these concerns:
- docdb (exists): the source of truth for document metadata and the catalog. MVCC, changes, replication primitives, and attachment (blob) storage.
- vectordb (exists): vector segments and local vector/text/hybrid search.
- barrel (exists): composes docdb and vectordb under one id space.
- server (exists,
barrel_server): transports over thebarrelAPI. - object storage: blobs are docdb attachments with a pluggable backend per
database (
barrel_att_backend: RocksDB BlobDB by default,nonefor databases without attachments, and S3-compatible storage throughbarrel_att_s3). There is no separate object-store app; a document owns its blob, matching how vectors and metadata attach to the same id. - agent layer (exists,
barrel_spaces+ MCP): spaces, capability tokens, sessions, and handoffs over thebarrelAPI; exposed throughbarrel_server. - contexts (exists,
barrel_ctx*inbarrel, REST and MCP inbarrel_server): named datasets queried together across nodes, with working sets and portable read-only copies. See the contexts guide. - fabric (future,
barrel_fabric): dataset agents, durability, placement, and replication orchestration across nodes.
Boundary rules
These rules keep the umbrella maintainable as it grows:
- Clear OTP application boundaries. Each app owns its modules and public API.
- No circular dependencies between apps. The direction is
barrel_server->barrel_spaces->barrel-> {barrel_docdb,barrel_vectordb},barrel_server->barrel_ngram->barrel_docdb, andbarrel_vectordb->barrel_embed(and optionallybarrel_faiss,barrel_rerank).barrel_docdbandbarrel_vectordbare leaves;barrel_docdb->barrel_vectordbis forbidden. - Prefer per-app modules over shared global modules. Avoid a common "junk drawer" application.
- Local use needs no external services. S3, remote replication, and the fabric agents are never required to run an app locally.
- Optional heavy backends (for example FAISS) stay opt-in and out of the default build.
Scope and rationale
This umbrella holds the Barrel data-stack libraries (barrel_docdb,
barrel_vectordb, barrel_embed, barrel_rerank, barrel_faiss,
barrel_ngram, barrel_crypto, barrel_att_s3), the embeddable database
(barrel), the agent layer (barrel_spaces), and the network server
(barrel_server). Products
and clients that build on top of the stack stay in their own repositories.
What the umbrella buys you
- Version coordination: one resolved dependency set across the apps. It already
forced a single
hackney(4.4.0) andmeck(1.2.0) where the separate repos had drifted, and surfaced the OTP 29 prefix-catchfix in one pass. - Atomic cross-app changes:
barrel_vectordband its backends (barrel_embed,barrel_rerank,barrel_faiss) co-evolve. A change to the vectordb API plus its adapters is one commit and one CI run, not tag-and-chase across repos. - One build, test, and release surface: shared profiles (the FAISS opt-in), one
PLT, one
rel/. - A home for the layers that span the stack:
barrel(the full database) already unifies docdb and vectordb under one id, andbarrel_fabric(future) will span them across nodes. A layer that unifies two apps needs a repo that holds both.barrel_docdbshares no code withbarrel_vectordb, so its place here is justified bybarreland the coming fabric work.
Out of scope (kept separate on purpose)
barrellm: an application/service layer that consumes the stack (barrellm->barrel_docdb+barrel_embed). It has its own HTTP/MCP surface, release cadence, and much older pinned deps (for example rocksdb 2.5.0, hackney 1.20.1/2.0.1). Folding it in would invert the layering and drag the stack back to older dependencies.barrel_llm: an empty placeholder directory. Nothing to import.
If products need co-development later
Keep them as separate repos that depend on the stack, via path deps for local
work ({path, "../barrel/apps/barrel_vectordb"}) or the published packages. If
they grow, group them in their own applications umbrella (for example
barrel_apps) that depends on this one, rather than flattening libraries and
products together.
Build layout
apps/holds the applications.- The top-level
rebar.configcoordinates the build and aggregates each app's declared dependencies. It restricts the default app set so the FAISS app is opt-in (rebar3 as faiss compile), as are the server (rebar3 as server compile) and the S3 backend (rebar3 as s3 compile). rel/holds the umbrella release skeleton.docs/holds umbrella-level documentation; per-app docs stay under each app.