# Embed barrel in an Erlang app

Source: https://github.com/barrel-db/barrel/blob/main/docs/guides/embedding.md

`barrel` is the embeddable edge database. It composes the document layer
(`barrel_docdb`) and the vector layer (`barrel_vectordb`) behind one API, where a
document, its attachments (blobs), and its vector share a single id. Read this
when you want a database inside your Erlang release without running a server.

## When to use it

- You want documents, vectors, BM25/hybrid search, attachments, and a changes
  feed from one handle, in-process.
- You do not need a network endpoint. For HTTP access, run `barrel_server` instead.

## Open and close

```erlang
{ok, _} = application:ensure_all_started(barrel_docdb),
{ok, _} = application:ensure_all_started(barrel_vectordb),

{ok, Db} = barrel:open(mydb),
%% pass options per layer:
{ok, Db2} = barrel:open(mydb, #{
    vectordb => #{dimension => 768, bm25_backend => memory}
}),
ok = barrel:close(Db).
```

A database links its vector store to the process that opened it. Open it from a
long-lived process (a gen_server or supervisor), not a transient one.

## Open without a vector store

A database that only holds documents does not need the vector store and its
RocksDB. Pass `vectordb => none`:

```erlang
{ok, Db} = barrel:open(notes, #{vectordb => none}),
{ok, _} = barrel:put_doc(Db, #{<<"id">> => <<"a">>, <<"lang">> => <<"en">>}),
{ok, _Rows, _Meta} = barrel:query(Db, "SELECT * FROM db WHERE lang = 'en'"),
{error, no_vector_store} = barrel:search(Db, <<"anything">>, #{}).
```

- Documents, attachments, queries, changes and replication work as usual.
- Vector calls (`vector_*`, `search*`, `embed*`, `embedder_info/1`,
  `vector_stats/1`) and the `vector_top_k`, `bm25_top_k` and `hybrid_top_k`
  table functions answer `{error, no_vector_store}`.
- Record mode needs a vector store: `embedding` with `vectordb => none` answers
  `{error, {invalid_option, vectordb}}`.
- A branch of such a database has no vector store either. A context export
  refuses it (`{error, no_vector_store}`); a slice of it holds documents only.
- `barrel_spaces:create_space(#{vectordb => none})` creates a space without one;
  the space reopens without one on every node.

## Open read only

Open with `read_only => true` to serve a database without writing any file,
for example an imported copy that several nodes share. Every write answers
`{error, read_only}`.

```erlang
{ok, Ro} = barrel:open(mydb, #{read_only => true}),
{error, read_only} = barrel:put_doc(Ro, #{<<"id">> => <<"z">>}),
{ok, _} = barrel:get_doc(Ro, <<"a">>),
ok = barrel:close(Ro).
```

- The database must exist: a missing one fails with
  `read_only_store_missing`. One written by an older version that needs an
  upgrade fails with `read_only_upgrade_needed`; open it writable once.
- Record mode persists no policy and starts no indexer. Pass
  `embedding => stored` to use the policy the database persisted
  (`{error, no_stored_policy}` on a plain database).

## Share handles through barrel_dbs

`barrel_dbs` caches open handles by name, closes idle ones and evicts past a
cap. Take a lease when a process needs a database to stay open while it
works; take a hold when you need its files closed (an export does).

```erlang
{error, not_found} = barrel_dbs:ensure(<<"nope">>, #{must_exist => true}),
{ok, Db, Lease} = barrel_dbs:lease(<<"mydb">>, #{}),
%% ... Db stays open against idle close and eviction ...
ok = barrel_dbs:release(Lease),

{ok, #{was_open := _}} = barrel_dbs:hold(<<"mydb">>, #{}),
%% ... every ensure of mydb is refused until unhold ...
ok = barrel_dbs:unhold(<<"mydb">>).
```

- Leases are counted and end when the holder exits; `barrel_dbs:leases/0`
  lists them. `lookup/1` returns the pinned flag, owner and open options.
- A hold is refused on a pinned or leased database, or one opened outside
  the manager.

## Documents

```erlang
{ok, _}   = barrel:put_doc(Db, #{<<"id">> => <<"a">>, <<"title">> => <<"hello">>}),
{ok, Doc} = barrel:get_doc(Db, <<"a">>),
{ok, _}   = barrel:delete_doc(Db, <<"a">>),
{ok, Rows, _Meta} = barrel:find(Db, #{where => [{path, [<<"title">>], <<"hello">>}]}).
```

## Batches

```erlang
[{ok, _}, {ok, _}] = barrel:put_docs(Db, [#{<<"id">> => <<"a">>}, #{<<"id">> => <<"b">>}]),
Results = barrel:get_docs(Db, [<<"a">>, <<"b">>]),     %% one result per id, in order
_       = barrel:delete_docs(Db, [<<"a">>, <<"b">>]).
```

## Attachments (blobs)

Blobs are document attachments; the storage backend is pluggable per database via
the docdb `barrel_att_backend` seam (RocksDB BlobDB by default; `barrel_att_s3`
stores them in an S3-compatible bucket).

```erlang
{ok, _}          = barrel:put_attachment(Db, <<"a">>, <<"f.txt">>, <<"bytes">>),
{ok, <<"bytes">>} = barrel:get_attachment(Db, <<"a">>, <<"f.txt">>),
[<<"f.txt">>]    = barrel:list_attachments(Db, <<"a">>),
ok               = barrel:delete_attachment(Db, <<"a">>, <<"f.txt">>).
```

Stream large blobs:

```erlang
{ok, W0} = barrel:open_attachment_writer(Db, <<"a">>, <<"big">>, <<"application/octet-stream">>),
{ok, W1} = barrel:write_attachment(W0, <<"chunk">>),
{ok, _}  = barrel:finish_attachment(W1),

{ok, R0} = barrel:open_attachment_reader(Db, <<"a">>, <<"big">>),
{ok, _Chunk, R1} = barrel:read_attachment(R0),    %% eof at the end
ok = barrel:close_attachment_reader(R1).
```

## Vectors and search

To keep vectors in sync with documents automatically, open the database with
an embedding policy instead of managing them by hand: see
[record mode](/docs/guides/record-mode). The direct vector API below applies to plain
databases.

```erlang
ok = barrel:vector_add(Db, <<"a">>, <<"hello world">>, #{}, Vector),
{ok, #{inserted := 2}} = barrel:vector_add_batch(Db, [
    {<<"a">>, <<"t1">>, #{}, V1},
    {<<"b">>, <<"t2">>, #{}, V2}
]),
{ok, Hits}  = barrel:search_vector(Db, Vector, #{k => 5}),
{ok, BHits} = barrel:search_bm25(Db, <<"hello">>, #{k => 5}).
```

Notes:

- `vector_add_batch/2` takes `{Id, Text, Metadata}` (text embedded by the store)
  or `{Id, Text, Metadata, Vector}` (explicit) tuples; a batch must be all one
  shape.
- BM25 is opt-in: open with `vectordb => #{bm25_backend => memory}` (or `disk`).
  A disk index survives a reopen; a memory index is rebuilt from the stored
  text at open.
- `search_hybrid/3` and auto-embedding adds need an embedder configured via
  `barrel_embed`; without one they return `{error, embedder_not_configured}`.
- To embed text yourself with the database's own embedder (same model and
  dimension as the indexed vectors), use `barrel:embed/2`,
  `barrel:embed_batch/2` and `barrel:embedder_info/1` on the handle; keep
  the handle opaque rather than reading its fields.

## Changes

```erlang
{ok, Changes, Last} = barrel:changes(Db, first),
Cursor  = barrel:hlc_encode(Last),         %% JSON/URL-safe cursor
Last2   = barrel:hlc_decode(Cursor),
{ok, More, _} = barrel:changes(Db, Last2),  %% changes since the cursor
{ok, Pid} = barrel:subscribe(Db, Last).
```
