CLIP Provider

Cross-modal embedding generation using CLIP models.

What is CLIP?

CLIP (Contrastive Language-Image Pre-training) encodes both images and text into the same vector space. This enables:

• Searching images with text queries
• Searching text with image queries
• Finding similar images
• Zero-shot image classification

Requirements

# Using virtualenv with uv (recommended)
./scripts/setup_venv.sh
uv pip install transformers torch pillow --python .venv/bin/python

# Or install manually
pip install transformers torch pillow

Configuration

{ok, State} = barrel_embed:init(#{
    embedder => {clip, #{
        python => "python3",                        % default, fallback only
        model => "openai/clip-vit-base-patch32",   % default
        timeout => 120000                           % default, ms
    }}
}).

Options

barrel_embed manages its own Python virtualenv automatically and installs transformers, torch, and pillow into it on first use. See Python Virtualenv Setup for the managed venv API.

Supported Models

API

Text Embedding

%% Text embeddings (same space as images)
{ok, TextVec} = barrel_embed:embed(<<"a photo of a cat">>, State).
{ok, TextVecs} = barrel_embed:embed_batch([<<"cat">>, <<"dog">>], State).

Image Embedding

Images must be base64-encoded:

%% Read and encode image
{ok, ImageData} = file:read_file("photo.jpg").
ImageBase64 = base64:encode(ImageData).

%% Get embedding
{ok, ImageVec} = barrel_embed_clip:embed_image(ImageBase64, Config).

%% Batch
{ok, ImageVecs} = barrel_embed_clip:embed_image_batch([Img1, Img2], Config).

Example: Image Search with Text

%% Initialize
{ok, State} = barrel_embed:init(#{embedder => {clip, #{}}}).
{_, Config} = hd(maps:get(providers, State)).

%% Index images (do once)
Images = [<<"img1.jpg">>, <<"img2.jpg">>, <<"img3.jpg">>],
ImageVecs = lists:map(fun(Path) ->
    {ok, Data} = file:read_file(Path),
    {ok, Vec} = barrel_embed_clip:embed_image(base64:encode(Data), Config),
    {Path, Vec}
end, Images).

%% Search with text query
{ok, QueryVec} = barrel_embed:embed(<<"a sunset over the ocean">>, State).

%% Find most similar images
Scores = [{Path, cosine_similarity(QueryVec, ImgVec)}
          || {Path, ImgVec} <- ImageVecs],
Ranked = lists:reverse(lists:keysort(2, Scores)).

Example: Similar Image Search

%% Find images similar to a reference image
{ok, RefData} = file:read_file("reference.jpg").
{ok, RefVec} = barrel_embed_clip:embed_image(base64:encode(RefData), Config).

%% Compare with other images
Scores = [{Path, cosine_similarity(RefVec, ImgVec)}
          || {Path, ImgVec} <- ImageVecs],
Similar = lists:reverse(lists:keysort(2, Scores)).

Example: Zero-Shot Classification

%% Classify an image into categories
Categories = [<<"a photo of a cat">>,
              <<"a photo of a dog">>,
              <<"a photo of a bird">>],
{ok, CategoryVecs} = barrel_embed:embed_batch(Categories, State).

{ok, ImageData} = file:read_file("mystery_animal.jpg"),
{ok, ImageVec} = barrel_embed_clip:embed_image(base64:encode(ImageData), Config).

%% Find best matching category
Scores = lists:zipwith(fun(Cat, CatVec) ->
    {Cat, cosine_similarity(ImageVec, CatVec)}
end, Categories, CategoryVecs),
[{BestCategory, _} | _] = lists:reverse(lists:keysort(2, Scores)).

Use Cases

• Image search: Find images by description
• Reverse image search: Find similar images
• Content moderation: Classify images automatically
• Multi-modal retrieval: Combined text and image search