Contexts and working sets
Data rarely sits in one database. A support team may keep European tickets on one server and American tickets on another, and an agent that answers "which tickets took the longest" needs both. Querying each server in turn and stitching the answers together is work, and when one server is down you also need to know what the answer is missing. Contexts let you ask once, by name, and get one answer that says what it covers.
A context is a named dataset, one Barrel database today, that lives on
this node, on another barrel_server,
or in an imported snapshot. You register a card for it, then run one
BQL statement over several contexts by
name and get one answer that says which contexts answered and what each
answer covers. A working set is the list of contexts you work with, some
of them saved locally so they answer offline. Read this to know the
model and its limits before you follow Query several
datasets.
The model
Each term below is one piece of that flow: cards say where data lives, a context query asks the question, and a working set keeps the contexts you need close at hand.
| Term | What it is |
|---|---|
| Card | A document in the node's catalog (_barrel_catalog): a name such as support/eu, an id ctx_..., one or more locations, and optional title, topics and embedding. Cards never hold secrets. |
| Location | local (a database on this node), remote (an endpoint and a database on another server) or snapshot (an imported export) |
| Context query | One BQL statement, the list of contexts (or a working_set), and how to merge the answers |
| Working set | A list of contexts. A member can be remote, a saved slice of query results, or an imported snapshot |
| Slice | The documents a query returned from one context, saved into a local database |
| Snapshot | A read-only copy of a database made with barrel_ctx_export:export/3 and imported on another node |
How a query runs
BQL has no syntax for contexts. You name them in the request, next to the
statement. Every member runs the same statement (the collection name
after FROM is ignored), in parallel, within budgets, and the node
merges the rows.
This query asks the two support contexts for their two longest tickets and merges them into one list, longest first:
{ok, R} = barrel_ctx:query(#{
query => <<"SELECT id, subject, minutes FROM c "
"ORDER BY minutes DESC LIMIT 2">>,
contexts => [<<"support/eu">>, <<"support/us">>]}).
The answer carries execution (succeeded, partial or failed),
rows or groups, one sources entry per context, coverage, and a
one-paragraph summary. A context that does not answer is reported in
its source entry, not as a failed request. If support/us is down, you
still get the European rows, with execution set to partial and the
reason in the support/us source entry.
Rules:
- A row query needs
LIMIT(at most 1000). AddORDER BYon a selected field to merge rows in order. bm25_top_kandhybrid_top_kanswers stay grouped per context; they are never merged into one ranking. BM25 scores depend on each database's own statistics, so scores from two contexts cannot be compared.- Refused:
SUBSCRIBE,OFFSET,UNNEST, continuations, and therrfandrerankmerges. - Default budgets: 8 contexts per query, a 5 s deadline (60 s at most), 4 s per member, 8 members in parallel (16 at most).
What is implemented today
Works today- The catalog: register, list, inspect, discover by topic, unregister.
- Context queries over local, remote and snapshot members, from Erlang
(
barrel_ctx), REST (/contexts/_query) and MCP (context_query). - Working sets: attach and detach members, materialize slices from a query, import snapshots, offline mode that skips remote members.
- Export and read-only import of snapshots.
- Credentials for remote servers in node configuration
(
ctx_credentials), never in cards.
What is planned
None of these exist yet. They are listed in the contexts design under "Not supported yet".
- Live federation (
SUBSCRIBEacross contexts). - BM25 and hybrid ranking across contexts, which needs shared BM25
statistics;
rrfandrerankmerges. - Enforced working-set ownership: the
ownerlabel is stored, not checked. Any global principal can read or delete any working set. - A query-only permission: a grant that allows querying a source but not copying from it.
- A shared or replicated catalog; today each node has its own, filled by hand.
- Publishing generations and ngram index packages to object storage.
- Connection reuse and TLS options in the remote client.
- Continuous subsets, predicate snapshots, pagination, and deduplication across replicas.
- REST and MCP routes to update a card (
barrel_ctx:update/2is Erlang only), and MCP tools to register and unregister.
Notes
barrel_ctx_export:export/3refuses a database with no vector store ({error, no_vector_store}).- An imported snapshot opens read only; it is not a restore.
- Registering, deleting and working-set routes need the global token on
barrel_server; querying is open to capability tokens, and each local member is then checked as a query on that database. - The design is in Contexts design.