barrel_docdb
An embedded document database on RocksDB. Writes are versioned with hybrid logical clocks and version vectors, every path is indexed for queries, and databases replicate active-active.
What it solves
Copies of the same data on several nodes, devices or agents get written while apart. barrel_docdb keeps every concurrent version as a conflict sibling instead of losing one, lets you resolve it, and gives you queries, a changes feed, attachments, retained history with provenance, TTL and branches on the same store.
When to use it
- You need documents that sync between nodes or sites, with conflicts kept and resolvable.
- You need queries on any field without declaring indexes, through
find/2or BQL. - You need an audit trail: who wrote each version, and the body of past versions.
When not to
- You also need vectors or semantic search: use barrel, which composes this with a vector store.
- You need aggregates, GROUP BY or joins: BQL has none.
How it fits into Barrel
- Builds on
barrel_crypto- Works with, when present
barrel_att_s3: stores attachment bytes in S3-compatible storage- Used by
barrel,barrel_ngram,barrel_spaces,barrel_server,barrel_att_s3- Other dependencies
- rocksdb, hackney, hlc, match_trie, mimerl, instrument
- On its own
- Yes. Replication (
barrel_rep,barrel_rep_tasks) and BQL for documents live here.
Install
{barrel_docdb, "~> 3.1"} 2.1.0 changed replication. Upgrading from 2.0 can need barrel_rep:reconcile/3.
Configuration
create_db/2 options
| Key | Default | What it does |
|---|---|---|
data_dir | app env data_dir | Where the database files go |
att_opts | RocksDB blob store | #{backend => none} skips it, s3 uses barrel_att_s3 |
encryption | disabled | default or #{provider => Mod} |
channels | none | #{Name => [Pattern]}, fixed at creation |
retention_period | 2592000 s (30 days) | History kept; 0 disables the sweeper |
ttl_sweep_interval | 0 (off) | Background sweep of expired documents |
max_conflict_versions | 8 | Conflict siblings kept per document |
conflict_resolver | none | Automatic resolution on replication |
App env (barrel_docdb)
| Key | Default | What it does |
|---|---|---|
data_dir | "data/barrel_docdb" | Relative to the current directory |
block_cache_size | 512 MiB | RocksDB block cache |
query_timeout_ms | 30000 | Query deadline |
The most used keys only. For every option, see the configuration reference and the API reference.
The API by task
Databases
| Task | Call | Returns |
|---|---|---|
| Create, open | barrel_docdb:create_db/1,2, open_db/1 | {ok, Pid} |
| Close, delete | barrel_docdb:close_db/1, delete_db/1,2 | ok |
| Describe, list | barrel_docdb:db_info/1, list_dbs/0 | info map, names |
Documents
| Task | Call | Returns |
|---|---|---|
| Write | barrel_docdb:put_doc/2,3, put_docs/2,3 | {ok, #{id, rev, ok}}; {error, conflict} without the current _rev |
| Read | barrel_docdb:get_doc/2,3, get_docs/2,3 | {ok, Doc} or {error, not_found} |
| Delete | barrel_docdb:delete_doc/2,3 | {ok, _} |
| Conflicts | barrel_docdb:get_conflicts/2, resolve_conflict/4 | {ok, [Rev]}, {ok, #{conflicts_resolved}} |
Query and changes
| Task | Call | Returns |
|---|---|---|
| Structured query | barrel_docdb:find/2,3, explain/2 | {ok, Rows, Meta} |
| BQL (documents only) | barrel_docdb:query/2,3 | {ok, Rows, Meta} |
| Changes feed | barrel_docdb:get_changes/2,3, subscribe_changes/2,3 | {ok, Changes, Last}, a stream pid |
| Path subscription | barrel_docdb:subscribe/2,3, unsubscribe/1 | {barrel_change, Db, Change} messages |
Files, history, timeline, sync
| Task | Call | Returns |
|---|---|---|
| Attachments | put_attachment/4,5, get_attachment/3, open_attachment_writer/4,5 | {ok, Info}, {ok, Bytes} |
| Versions | get_doc_versions/2, get_version_body/3, fold_history/3,4 | {ok, [Version]} with provenance |
| Branch, merge | branch_db/3, merge_branch/2, list_branches/1 | {ok, _}, {ok, Report} |
| Replicate once | barrel_rep:replicate/2,3 | {ok, Report} |
| Continuous task | barrel_rep_tasks:start_task/1, get_task/1, stop_task/1 | {ok, TaskId} |
Write, find and query
From a verified run on a fresh data directory.
{ok, _} = application:ensure_all_started(barrel_docdb).
{ok, _} = barrel_docdb:create_db(<<"kb">>).
barrel_docdb:put_docs(<<"kb">>, [
#{<<"id">> => <<"kb:101">>, <<"type">> => <<"article">>,
<<"title">> => <<"Reset a forgotten password">>,
<<"views">> => 1200,
<<"tags">> => [<<"account">>, <<"login">>]},
#{<<"id">> => <<"kb:102">>, <<"type">> => <<"article">>,
<<"title">> => <<"Configure SSO with Okta">>,
<<"views">> => 430,
<<"tags">> => [<<"sso">>, <<"login">>]},
#{<<"id">> => <<"kb:201">>, <<"type">> => <<"article">>,
<<"title">> => <<"Export invoices as CSV">>,
<<"views">> => 980,
<<"tags">> => [<<"export">>]}]).
barrel_docdb:find(<<"kb">>, #{id_prefix => <<"kb:1">>,
include_docs => false}).
barrel_docdb:query(<<"kb">>,
<<"SELECT title, views FROM kb WHERE type = 'article' "
"AND views >= 500 ORDER BY views DESC">>). [{ok,#{<<"id">> => <<"kb:101">>,<<"ok">> => true,
<<"rev">> =>
<<"000001a1b8784bea00000000@waec6d4b23aba0dca">>}},
...]
{ok,[#{<<"id">> => <<"kb:101">>},#{<<"id">> => <<"kb:102">>}],
#{has_more => false, ...}}
{ok,[#{<<"id">> => <<"kb:101">>,
<<"title">> => <<"Reset a forgotten password">>,
<<"views">> => 1200},
#{<<"id">> => <<"kb:201">>,
<<"title">> => <<"Export invoices as CSV">>,
<<"views">> => 980}],
#{count => 2, has_more => false, ...}} Resolve a conflict after two-way replication
Two local databases edited the same document, then replicated both ways. The resolution names the versions it observed.
barrel_rep:replicate(<<"edge_a">>, <<"edge_b">>),
barrel_rep:replicate(<<"edge_b">>, <<"edge_a">>).
%% both sides edited task:1 while apart
barrel_docdb:get_conflicts(<<"edge_a">>, <<"task:1">>).
{ok, #{<<"_observed">> := Seen}} =
barrel_docdb:get_doc(<<"edge_a">>, <<"task:1">>,
#{conflicts => true}).
barrel_docdb:resolve_conflict(<<"edge_a">>, <<"task:1">>,
Seen, {merge, Merged}).
%% Merged: the body to keep, here with a "note" field {ok,[<<"000001a1b879d59100000000@w7bcb971434496877">>]}
{ok,#{id => <<"task:1">>,
rev => <<"000001a1b879d59500000000@w7bcb971434496877">>,
conflicts_resolved => 1}} Limits to know
- Lineage is linear: a branch of a branch is refused, and a point-in-time branch older than the history floor fails with
pitr_window_exceeded. - Replication does not carry vectors or provenance. Channels do not delete departed documents on replicas.
- Replication tasks refuse inline credentials. Attachment checkpoints wait for the target to be durable unless you set
att_durability => best_effort. - Runtime settings (
encryption,read_only,channels,conflict_resolver, retention) must be passed again on every reopen. - The app env
data_diris relative to the current directory. Thecreate_db/2doc still says/tmp/barrel_data.
find/2 order_by on a path is ignored unless that path is selected or bound to a variable.
A chunked BQL query with a range predicate dropped a row on continuation once in testing; not yet minimized.