Barrel
FR GitHub

Substring and regex

barrel_ngram finds exact substrings and PCRE regular expressions in the documents of a barrel_docdb database. It is a trigram index kept up to date from the database's changes feed, and every candidate is checked against the real text, so a hit is always a real match. Use it for identifiers, error codes, config keys and file paths: the strings that semantic and keyword search miss or split apart.

Add it

{deps, [{barrel_ngram, "~> 0.11"}]}.

It builds a native library with CMake. The barrel API and BQL do not expose it: you call barrel_ngram directly, or the ngram_search MCP tool on barrel_server.

Open an index

An index is a corpus over one database. Name the fields to index, or leave fields => all.

{ok, _} = application:ensure_all_started(barrel_ngram),
{ok, _} = barrel_docdb:create_db(<<"runbooks">>),
barrel_docdb:put_docs(<<"runbooks">>, [
  #{<<"id">> => <<"rb:db">>,
    <<"title">> => <<"Postgres failover">>,
    <<"body">> => <<"If you see ERR_CONN_TIMEOUT=30s on "
                    "pg-primary-01, promote pg-replica-02.">>},
  #{<<"id">> => <<"rb:cache">>,
    <<"title">> => <<"Redis eviction">>,
    <<"body">> => <<"Set maxmemory-policy allkeys-lru; watch "
                    "evicted_keys in INFO stats.">>},
  #{<<"id">> => <<"rb:api">>, <<"title">> => <<"API 502s">>,
    <<"body">> => <<"Check upstream connect_timeout and "
                    "read_timeout in nginx.conf">>}]),
ok = barrel_ngram:open(<<"rb_idx">>,
                       #{db => <<"runbooks">>,
                         fields => [<<"title">>, <<"body">>]}),
barrel_ngram:index(<<"rb_idx">>).

index/1 (also refresh/1) catches the index up with the feed and returns. After that, new writes reach the index in the background; call it again when you need the latest writes to be searchable.

barrel_ngram:search(<<"rb_idx">>, <<"ERR_CONN_TIMEOUT">>),
barrel_ngram:regex(<<"rb_idx">>,
                 <<"pg-(primary|replica)-\\d+">>),
barrel_ngram:regex(<<"rb_idx">>, <<"\\w+_timeout">>).
Output from a run on barrel_ngram 0.11.4 (Erlang)
{ok,[#{id => <<"rb:db">>,spans => [{11,16}]}]}
{ok,[#{id => <<"rb:db">>,spans => [{35,13},{58,13}]}]}
{ok,[#{id => <<"rb:api">>,spans => [{15,15},{35,12}]}]}

A hit is the document id and the matches as {Offset, Length} in bytes. Offsets count in the indexed text, which joins the indexed fields in sorted field order (body before title).

Case-insensitive search:

barrel_ngram:search(<<"rb_idx">>, <<"redis">>,
                    #{case_sensitive => false}).
Output from a run on barrel_ngram 0.11.4
{ok,[#{id => <<"rb:cache">>,spans => [{68,5}]}]}

A bad pattern returns {error, {bad_regex, _}}.

Over MCP

ngram_search takes db, query, mode (literal by default, or regex) and limit (50 by default, 1000 at most). The server opens one index per database on first use, over every field, under its data directory, and refreshes it before each search.

Output from a run on barrel_server 1.13.0: literal query 'over MCP' on a database holding one note
data: {"id":11,"jsonrpc":"2.0","result":{"content":[{"text":
"{\"count\":1,\"hits\":[{\"id\":\"note-1\",
\"spans\":[[7,8]]}]}","type":"text"}]}}

Maintain it

  • compact/1 merges segments; it returns {error, busy} if a merge is running.
  • close/1 closes the index; open/2 with the same options reopens it.
  • delete_corpus/1 removes it.
  • db, fields, shards, postings and phase2_selector_opts are fixed for the life of a corpus. Opening with another value returns {error, {config_mismatch, Field, Persisted, Requested}}; delete the corpus and rebuild.
  • An index written by 0.10 or older needs on_legacy => reindex.
Limit

The index takes about 34 times the size of the indexed text. Index only the fields you search. Data goes to data/barrel_ngram under the current directory unless you set the data_dir app env or option.

Written for the site Edit this page View as Markdown

Up and down to moveEnter to open