barrel_faiss
Erlang bindings to the FAISS C++ library, and the optional faiss index backend of barrel_vectordb.
What it solves
Some workloads want FAISS index types (IVF, PQ, flat, FAISS HNSW) instead of the pure Erlang HNSW. barrel_faiss wraps FAISS as a NIF so barrel_vectordb can use it, and you can also call it directly.
When to use it
- You already rely on FAISS index types or their memory and speed trade-offs.
- You can install the FAISS library on every node that opens the store.
When not to
- You want a build with no system libraries: the default HNSW backend is pure Erlang.
- You build on Windows: there are no build hooks for it.
How it fits into Barrel
- Builds on
- No other Barrel library
- Used by
- No other Barrel library depends on it
- Optional in
barrel_vectordb- Other dependencies
- FAISS library and headers, CMake 3.14 or later, C++17, OpenMP, BLAS
- On its own
- Yes, as NIF bindings with no Erlang dependencies. It needs FAISS installed on the system.
Install
{barrel_faiss, "~> 1.0"} In the umbrella it is not in the default build: use rebar3 as faiss compile.
Configuration
barrel_vectordb store config
| Key | Default | What it does |
|---|---|---|
backend | hnsw | Set faiss |
faiss.index_type | <<"HNSW32">> | A FAISS factory string |
faiss.nprobe | none | For IVF indexes |
The most used keys only. For every option, see the configuration reference and the API reference.
The API by task
Index
| Task | Call | Returns |
|---|---|---|
| Create | barrel_faiss:new/1,2, index_factory/2,3 | {ok, Index}; metric l2 or inner_product |
| Train (IVF, PQ) | barrel_faiss:train/2, is_trained/1 | ok, boolean |
| Add | barrel_faiss:add/2, add_with_ids/3 | ok; float32 and int64, native endian |
| Search | barrel_faiss:search/3 | {ok, Distances, Labels} |
| Remove | barrel_faiss:remove_ids/2 | {ok, N}; not on HNSW |
| Save, load | serialize/1, deserialize/1, write_index/2, read_index/1 | binary or file |
| Inspect, close | dimension/1, ntotal/1, close/1 | integers, ok |
Direct calls
From a verified run on a fresh CMake build against FAISS from MacPorts. The run printed the values below; the print statements are left out of the code.
{ok, Idx} = barrel_faiss:new(4),
Vecs = << <<X:32/float-native>> || X <-
[1.0,0.0,0.0,0.0, 0.0,1.0,0.0,0.0,
0.0,0.0,1.0,0.0] >>,
ok = barrel_faiss:add(Idx, Vecs),
Q = << <<X:32/float-native>> || X <- [0.9,0.1,0.0,0.0] >>,
{ok, D, L} = barrel_faiss:search(Idx, Q, 2),
{ok, Bin} = barrel_faiss:serialize(Idx),
{ok, H} = barrel_faiss:index_factory(4, <<"IDMap,Flat">>,
inner_product),
ok = barrel_faiss:add_with_ids(H, Vecs,
<< <<I:64/signed-native>> || I <- [100, 200, 300] >>),
{ok, _, L2} = barrel_faiss:search(H, Q, 1),
barrel_faiss:remove_ids(H, <<100:64/signed-native>>),
ok = barrel_faiss:close(Idx). ntotal=3 dim=4 trained=true
distances=[0.020000005140900612,1.619999885559082] labels=[0,1]
deserialized ntotal=3 (93 bytes)
IDMap top label=[100]
remove_ids={ok,1} ntotal=2
after close: {error,"index_closed"} As the backend of a barrel vector store
The same run, through barrel. The last line is the same open on a build without the NIF.
{ok, Db} = barrel:open(fdb, #{
docdb => #{data_dir => Dir},
vectordb => #{dimension => 3, backend => faiss,
db_path => Dir ++ "/fdb_vec"}}),
ok = barrel:vector_add(Db, <<"a">>, <<"alpha">>, #{},
[1.0, 0.0, 0.0]),
ok = barrel:vector_add(Db, <<"b">>, <<"beta">>, #{},
[0.0, 1.0, 0.0]),
{ok, Hits} = barrel:search_vector(Db, [0.9, 0.1, 0.0],
#{k => 1}). faiss-backed hit: [<<"a">>]
without NIF: {error,{index_init_failed,faiss_not_available}} Limits to know
- Needs a system FAISS; the build looks in FAISS_ROOT, /opt/local, /usr/local, /opt/homebrew and /usr.
- Runs on dirty schedulers.
remove_ids/2is not supported on HNSW indexes. - A search on a closed index returns
{error, "index_closed"}, a string rather than an atom.