barrel_vectordb_turboquant_subspace (barrel_vectordb v2.1.1)
View SourceSubspace-TurboQuant: O(D/M) scalable vector quantization
Addresses O(D^2) scaling issues in standard TurboQuant by splitting D-dimensional vectors into M independent subspaces. Each subspace applies TurboQuant with its own rotation matrix, reducing:
- Rotation matrix memory: D^2 -> M * (D/M)^2 = D^2/M (8x reduction for M=8) - Encode latency: O(D^2) -> O(D^2/M) per subspace, parallelizable - No training required (preserves data-oblivious property)
Storage format: Header: <<Version:8, Bits:8, M:8, Flags:8>> Body: [SubspaceCode_1, ..., SubspaceCode_M] Each subspace code contains radii, angles, and QJL signs
Performance for D=768, M=8: - Rotation matrices: 8 * 96^2 * 8 = 590KB (vs 4.7MB) - Encode latency: ~0.5ms (vs ~3.6ms) - Recall: Within 2-3% of full TurboQuant
Summary
Functions
Batch compute ADC distance using SIMD-accelerated NIF
Batch encode multiple vectors
Decode Subspace-TurboQuant code back to approximate vector
Compute asymmetric distance using precomputed tables (pure Erlang)
Compute ADC distance using SIMD-accelerated NIF
Encode a vector using Subspace-TurboQuant Returns compact binary with header + M subspace codes
Get configuration info
Create a new Subspace-TurboQuant configuration Options: bits - bits per component (default: 3, range: 2-4) dimension - vector dimension (required, must be even and divisible by m) m - number of subspaces (default: auto-selected based on dimension) seed - base random seed (default: 42)
Precompute distance lookup tables for a query vector Returns M sets of tables (concatenated)
Auto-select M based on dimension for optimal performance Keeps subdim around 64-128 for best SIMD performance
Types
-type distance_tables() :: binary().
-type tq_subspace_code() :: binary().
-type tq_subspace_config() :: #tq_subspace_config{bits :: 2..4, m :: pos_integer(), dimension :: pos_integer(), subdim :: pos_integer(), subspace_configs :: [term()], seeds :: [integer()]}.
Functions
-spec batch_distance_nif(distance_tables(), [tq_subspace_code()]) -> [float()].
Batch compute ADC distance using SIMD-accelerated NIF
-spec batch_encode(tq_subspace_config(), [[float()]]) -> [tq_subspace_code()].
Batch encode multiple vectors
-spec decode(tq_subspace_config(), tq_subspace_code()) -> [float()].
Decode Subspace-TurboQuant code back to approximate vector
-spec distance(distance_tables(), tq_subspace_code()) -> float().
Compute asymmetric distance using precomputed tables (pure Erlang)
-spec distance_nif(distance_tables(), tq_subspace_code()) -> float().
Compute ADC distance using SIMD-accelerated NIF
-spec encode(tq_subspace_config(), [float()]) -> tq_subspace_code().
Encode a vector using Subspace-TurboQuant Returns compact binary with header + M subspace codes
-spec info(tq_subspace_config()) -> map().
Get configuration info
-spec new(map()) -> {ok, tq_subspace_config()} | {error, term()}.
Create a new Subspace-TurboQuant configuration Options: bits - bits per component (default: 3, range: 2-4) dimension - vector dimension (required, must be even and divisible by m) m - number of subspaces (default: auto-selected based on dimension) seed - base random seed (default: 42)
-spec precompute_tables(tq_subspace_config(), [float()]) -> distance_tables().
Precompute distance lookup tables for a query vector Returns M sets of tables (concatenated)
-spec select_m(pos_integer()) -> pos_integer().
Auto-select M based on dimension for optimal performance Keeps subdim around 64-128 for best SIMD performance
-spec split_subvectors([float()], pos_integer(), pos_integer()) -> [[float()]].