Barrel
FR GitHub

barrel

The embeddable database. One barrel database is a document database and a vector store under one name and one id space, so a document, its attachments and its vector share an id.

What it solves

Agents and search features need documents, files and vectors that stay together. Keeping a document store and a vector index side by side means two ids, two lifecycles and writes that drift apart. barrel composes them, adds record mode (embeddings kept in sync with the documents), BQL with search table functions and live queries, timeline (branch, recover, merge), contexts and export.

When to use it

  • You build an Erlang or Elixir application and want documents, attachments and vector search in-process.
  • You want embeddings computed from document fields and kept in sync on every write (record mode).
  • You want to branch a database for an agent run, review it, and merge it back.

When not to

  • You need clients over the network: run barrel_server on top.
  • You only need documents and sync, no vectors: barrel_docdb alone is enough.
  • You need replication through this facade: it is not exposed here yet; call barrel_rep on the docdb name.

How it fits into Barrel

Other dependencies
hackney (remote context client); OTP 26 or later
On its own
Yes, as a library. It serves no network protocol on its own.

Install

rebar.config deps
{barrel, "~> 1.14"}

Record mode needs the barrel application started; otherwise opens with a policy return {error, barrel_app_not_started}.

Configuration

barrel:open/2 options

KeyDefaultWhat it does
docdb #{} Options passed to barrel_docdb:create_db/2
vectordb a vector store Store config map, or none for documents only
vectordb.db_path <data_dir>/<name>.vectordb Next to the documents. A store created before 1.13.0 in ./priv/barrel_vectordb_<name> stays there.
embedding none Record-mode policy map, or stored to reuse the saved one
encryption disabled default or #{provider => Mod}
read_only false Writes return {error, read_only}
store_supervised false By default the vector store is linked to the caller

Record-mode policy

KeyDefaultWhat it does
fields all Paths whose text is embedded
mode async sync embeds before the write returns
dimensions 768 Vector size
embedder none A barrel_embed provider config
metadata_fields document minus id and _ keys Fields copied into vector metadata

App env (barrel)

KeyDefaultWhat it does
dbs_idle_timeout 300000 ms barrel_dbs closes idle databases; 0 disables
dbs_max_open 0 Open database limit; 0 is unlimited
ctx_max_contexts 8 Contexts per federated query
ctx_deadline_ms 5000 Context query deadline (max 60000)

The most used keys only. For every option, see the configuration reference and the API reference.

The API by task

Open and manage

TaskCallReturns
Open or create barrel:open/1,2 {ok, Db}
Close, delete barrel:close/1, delete/1 ok
Describe barrel:info/1 docdb info plus embedding, dimensions, embedder
Managed lifecycle barrel_dbs:ensure/1,2, lease/2, release/1 {ok, Db}, {ok, Db, LeaseRef}

Documents and files

TaskCallReturns
Write barrel:put_doc/2,3, put_docs/2,3 {ok, #{<<"id">>, <<"rev">>, <<"ok">>}}
Read barrel:get_doc/2,3, get_docs/2,3 {ok, Doc}
Delete barrel:delete_doc/2,3 {ok, _}
Find barrel:find/2,3 {ok, Rows, Meta}
Attach a file barrel:put_attachment/4,5, get_attachment/3 {ok, Info}, {ok, Bytes}

Search and query

TaskCallReturns
Add a vector barrel:vector_add/4,5, vector_add_batch/2 ok, {ok, #{inserted}}
Semantic search barrel:search/3, search_vector/3 {ok, [#{key, score, text, metadata}]}
Keyword, hybrid barrel:search_bm25/3, search_hybrid/3 {ok, [{Id, Score}]}, {ok, [Hit]}
BQL barrel:query/2,3, explain_query/2,3 {ok, Rows, Meta}
Live query barrel:subscribe_query/2,3, unsubscribe_query/1 {ok, #{pid, ref}}, then bql_rows, bql_ready, bql_change messages

Timeline and history

TaskCallReturns
Branch barrel:branch/2,3 {ok, BranchDb}
Merge back barrel:merge/1,2 {ok, Report}
Changes barrel:changes/2,3, subscribe/2,3 changes list, or a stream pid
Versions barrel:doc_versions/2, version_body/3 {ok, [Version]}, {ok, Body}

Documents, vectors, BM25 and BQL in one database

From a verified run. kb_embed is a 64-dimension demo provider written for the run (feature hashing: it matches shared words, it is not semantic). Any barrel_embed provider works in its place.

Erlang shell
{ok, _} = application:ensure_all_started(barrel).
{ok, Db} = barrel:open(<<"research">>, #{vectordb => #{
    db_path => filename:join(VecDir, "research"),
    dimensions => 64,
    embedder => {kb_embed, #{}},
    bm25_backend => memory}}).
Notes = [
  {<<"paper:raft">>, <<"Raft consensus: leader election "
     "and log replication for replicated state machines">>,
   #{<<"topic">> => <<"distributed-systems">>,
     <<"year">> => 2014}},
  {<<"paper:crdt">>, <<"Conflict-free replicated data "
     "types converge without coordination">>,
   #{<<"topic">> => <<"distributed-systems">>,
     <<"year">> => 2011}},
  {<<"paper:hnsw">>, <<"HNSW: hierarchical navigable small "
     "world graphs for approximate nearest neighbor "
     "search">>,
   #{<<"topic">> => <<"vector-search">>,
     <<"year">> => 2016}},
  {<<"paper:bm25">>, <<"Okapi BM25 ranking function for "
     "keyword search over documents">>,
   #{<<"topic">> => <<"information-retrieval">>,
     <<"year">> => 1994}}].
[{ok, _} = barrel:put_doc(Db,
    M#{<<"id">> => Id, <<"abstract">> => T})
 || {Id, T, M} <- Notes].
barrel:vector_add_batch(Db, Notes).
barrel:search(Db, <<"leader election in replicated logs">>,
              #{k => 2}).
barrel:search_bm25(Db, <<"keyword search">>, #{k => 2}).
barrel:query(Db, <<"SELECT v.id, v._score, v._distance, "
    "abstract FROM vector_top_k("
    "'replicated state machines', k => 2) AS v">>).
Output, from a run on barrel 1.12.0 (trimmed)
{ok,#{inserted => 4}}

{ok,[#{metadata => #{<<"topic">> => <<"distributed-systems">>,
                     <<"year">> => 2014},
       text => <<"Raft consensus: ...">>,
       key => <<"paper:raft">>,score => 0.5257737534501699},
     #{metadata => #{<<"topic">> => <<"distributed-systems">>,
                     <<"year">> => 2011},
       text => <<"Conflict-free replicated data types ...">>,
       key => <<"paper:crdt">>,score => 0.14173227881546058}]}

{ok,[{<<"paper:bm25">>,1.9587592623731485},
     {<<"paper:hnsw">>,0.6586049622596158}]}

{ok,[#{<<"_distance">> => 0.5158329928408527,
       <<"_score">> => 0.4841670071591473,
       <<"abstract">> => <<"Raft consensus: ...">>,
       <<"id">> => <<"paper:raft">>},
     #{<<"_distance">> => 0.7817821172596275,
       <<"_score">> => 0.21821788274037246,
       <<"abstract">> => <<"Conflict-free replicated ...">>,
       <<"id">> => <<"paper:crdt">>}],
    #{count => 2, has_more => false, ...}}

Limits to know

  • A vector store created before barrel 1.13.0 without db_path stays in ./priv/barrel_vectordb_<name> under the current directory, with a notice in the log. A backup of data_dir alone misses it; new stores go to <data_dir>/<name>.vectordb.
  • The vector store is linked to the process that opened it. Use barrel_dbs or store_supervised => true for long-lived owners.
  • In plain mode, put_doc and vector_add are separate writes. Record mode is the atomic path.
  • Changing a record-mode policy on reopen logs a warning and does not reindex existing documents.
  • Plain branches carry no vectors. A branch of a branch is refused (cannot_branch_a_branch).
  • Export needs a vector store ({error, no_vector_store} otherwise) and closes the database during the copy.
  • Context queries need LIMIT; OFFSET, UNNEST, SUBSCRIBE and continuation are refused.
Known gap

find/2 order_by on a path is ignored unless the path is selected or bound to a variable. BQL ORDER BY sorts correctly.

Up and down to moveEnter to open