Barrel
FR GitHub

barrel_faiss

Erlang bindings to the FAISS C++ library, and the optional faiss index backend of barrel_vectordb.

What it solves

Some workloads want FAISS index types (IVF, PQ, flat, FAISS HNSW) instead of the pure Erlang HNSW. barrel_faiss wraps FAISS as a NIF so barrel_vectordb can use it, and you can also call it directly.

When to use it

  • You already rely on FAISS index types or their memory and speed trade-offs.
  • You can install the FAISS library on every node that opens the store.

When not to

  • You want a build with no system libraries: the default HNSW backend is pure Erlang.
  • You build on Windows: there are no build hooks for it.

How it fits into Barrel

Builds on
No other Barrel library
Used by
No other Barrel library depends on it
Optional in
barrel_vectordb
Other dependencies
FAISS library and headers, CMake 3.14 or later, C++17, OpenMP, BLAS
On its own
Yes, as NIF bindings with no Erlang dependencies. It needs FAISS installed on the system.

Install

rebar.config deps
{barrel_faiss, "~> 1.0"}

In the umbrella it is not in the default build: use rebar3 as faiss compile.

Configuration

barrel_vectordb store config

KeyDefaultWhat it does
backend hnsw Set faiss
faiss.index_type <<"HNSW32">> A FAISS factory string
faiss.nprobe none For IVF indexes

The most used keys only. For every option, see the configuration reference and the API reference.

The API by task

Index

TaskCallReturns
Create barrel_faiss:new/1,2, index_factory/2,3 {ok, Index}; metric l2 or inner_product
Train (IVF, PQ) barrel_faiss:train/2, is_trained/1 ok, boolean
Add barrel_faiss:add/2, add_with_ids/3 ok; float32 and int64, native endian
Search barrel_faiss:search/3 {ok, Distances, Labels}
Remove barrel_faiss:remove_ids/2 {ok, N}; not on HNSW
Save, load serialize/1, deserialize/1, write_index/2, read_index/1 binary or file
Inspect, close dimension/1, ntotal/1, close/1 integers, ok

Direct calls

From a verified run on a fresh CMake build against FAISS from MacPorts. The run printed the values below; the print statements are left out of the code.

Erlang
{ok, Idx} = barrel_faiss:new(4),
Vecs = << <<X:32/float-native>> || X <-
          [1.0,0.0,0.0,0.0, 0.0,1.0,0.0,0.0,
           0.0,0.0,1.0,0.0] >>,
ok = barrel_faiss:add(Idx, Vecs),
Q = << <<X:32/float-native>> || X <- [0.9,0.1,0.0,0.0] >>,
{ok, D, L} = barrel_faiss:search(Idx, Q, 2),
{ok, Bin} = barrel_faiss:serialize(Idx),
{ok, H} = barrel_faiss:index_factory(4, <<"IDMap,Flat">>,
                                     inner_product),
ok = barrel_faiss:add_with_ids(H, Vecs,
    << <<I:64/signed-native>> || I <- [100, 200, 300] >>),
{ok, _, L2} = barrel_faiss:search(H, Q, 1),
barrel_faiss:remove_ids(H, <<100:64/signed-native>>),
ok = barrel_faiss:close(Idx).
Printed by the run, barrel_faiss 1.0.1
ntotal=3 dim=4 trained=true
distances=[0.020000005140900612,1.619999885559082] labels=[0,1]
deserialized ntotal=3 (93 bytes)
IDMap top label=[100]
remove_ids={ok,1} ntotal=2
after close: {error,"index_closed"}

As the backend of a barrel vector store

The same run, through barrel. The last line is the same open on a build without the NIF.

Erlang
{ok, Db} = barrel:open(fdb, #{
    docdb => #{data_dir => Dir},
    vectordb => #{dimension => 3, backend => faiss,
                  db_path => Dir ++ "/fdb_vec"}}),
ok = barrel:vector_add(Db, <<"a">>, <<"alpha">>, #{},
                       [1.0, 0.0, 0.0]),
ok = barrel:vector_add(Db, <<"b">>, <<"beta">>, #{},
                       [0.0, 1.0, 0.0]),
{ok, Hits} = barrel:search_vector(Db, [0.9, 0.1, 0.0],
                                  #{k => 1}).
Printed by the run
faiss-backed hit: [<<"a">>]
without NIF: {error,{index_init_failed,faiss_not_available}}

Limits to know

  • Needs a system FAISS; the build looks in FAISS_ROOT, /opt/local, /usr/local, /opt/homebrew and /usr.
  • Runs on dirty schedulers. remove_ids/2 is not supported on HNSW indexes.
  • A search on a closed index returns {error, "index_closed"}, a string rather than an atom.

Up and down to moveEnter to open