Monitoring
This page covers what Barrel measures, how you read it, and how you watch replication tasks. Read it when you put a node in production and need to know it is healthy and keeping up.
What Barrel measures
The document layer, barrel_docdb, records metrics on every node, whether or not you export them. They follow Prometheus conventions: a counter ends in _total, and labels are exported as labels. Write your queries against the names below.
| Exported name | Type | What it counts |
|---|---|---|
barrel_doc_operations_total{db,operation} | counter | Document puts, gets and deletes |
barrel_doc_operation_duration_seconds{db,operation} | histogram | Time per document operation |
barrel_write_group_size{db} | histogram | Writes committed together in one batch |
barrel_query_operations_total{db} | counter | Queries run |
barrel_query_duration_seconds{db} | histogram | Time per query |
barrel_query_results_count{db} | histogram | Rows returned per query |
barrel_query_timeouts_total | counter | Queries that hit their deadline |
barrel_replication_docs_total{direction} | counter | Documents written by replication |
barrel_replication_errors_total{task_id} | counter | Replication errors, per replication id |
barrel_replication_active{task_id} | gauge | 1 while the task runs, 0 when it is paused, failed, completed or deleted |
barrel_replication_lag_seconds{task_id} | gauge | Seconds since the last copied change was written on the source; 0 once caught up |
barrel_db_documents_total{db} | gauge | Live documents (deleted ones are not counted), exact |
barrel_db_attachments_total{db} | gauge | Attachments, exact, for both the blob and S3 backends |
barrel_db_size_bytes{db} | gauge | Document store size on disk |
barrel_db_pending_compaction_bytes{db} | gauge | Bytes compaction still has to rewrite |
barrel_db_compactions_running{db}, barrel_db_compactions_queued{db} | gauge | Compactions running now, and waiting for a thread |
barrel_db_stubs_total{db}, barrel_db_stubs_bytes{db} | gauge | Deleted documents kept as stubs, after each retention sweep |
The size and compaction gauges refresh every minute (stats_interval), and once within a second of opening. A pending backlog that keeps growing, or queued compactions that stay above zero, means compaction is falling behind your writes.
A metric appears in the export once it has a value: a node that never replicated prints no replication metrics.
Before barrel_docdb 3.0 the labels were folded into the name: document operations were exported as barrel_doc_operations_db_operation_total{db,operation}, and the same went for every labeled metric. Update dashboards and alerts that use the old names.
The document and attachment counts are exact: every write that adds or removes one updates the count in the same atomic write (barrel_docdb 2.4.0). A database written before 2.4.0 is counted once in the background after it opens; until then its counts are not published and db_info reports them as undefined.
barrel_vectordb, barrel, barrel_server, barrel_spaces and barrel_embed record no metrics.
Read the metrics
Inside the node, barrel_metrics:export_text/0 returns the Prometheus text format as a binary (export/0 returns iodata).
io:format("~s", [barrel_metrics:export_text()]).
barrel_doc_operations_total{db="mon",operation="put"} 1.0
barrel_doc_operations_total{db="mon",operation="get"} 1.0
barrel_query_operations_total{db="mon"} 1.0
barrel_doc_operation_duration_seconds_count{db="mon",operation="put"} 1 Send them to a collector
barrel_server does not serve a /metrics route. Push the metrics over OTLP to your collector, which Prometheus or another backend then reads. Set the exporters in the barrel_docdb application environment:
[{barrel_docdb, [
{metrics, [{exporter, otlp},
{otlp_endpoint, "http://collector:4318"}]},
{tracing, [{enabled, true}, {exporter, otlp},
{otlp_endpoint, "http://collector:4318"}]}
]}].
The exporter is none by default; console and {Mod, Map} are also accepted. Metrics, tracing and logging each take their own exporter, see Configuration.
Tracing covers document operations, queries and replication. Outgoing replication requests carry trace headers to the peer.
The export_interval key shown in some barrel_docdb docs is never read.
If you want a scrape endpoint, serve barrel_metrics:export_text() from a handler in your own application.
Stats you can ask for
| Call | Returns |
|---|---|
barrel:info(Db), GET /db/:db | Configuration, retention, history floor, write-group counters, embedder, and storage: size on disk, compaction backlog, and the documents and attachments counts. |
barrel:vector_stats(Db) | Vector count, dimension, index type and whether the index was loaded or rebuilt |
barrel_docdb:stub_stats(Name) | Stub count and bytes. Scans the change feed in the caller. |
barrel_rep_tasks:list_tasks() | Every replication task with its status |
barrel_rep_tasks:get_task(Id) | One task: status, last_seq, last_error, paused_reason |
{ok, S} = barrel:vector_stats(Db),
maps:with([count, dimension, index_origin], S).
The MCP tool db_info says it returns a document count; it returns the same map as barrel:info/1, which has none.
Watch replication tasks
A continuous task keeps the status running while it retries, with exponential backoff. The error is on the task, not in its status, so alert on last_error.
{ok, Tasks} = barrel_rep_tasks:list_tasks(#{mode => continuous}),
[{Id, Status, maps:get(last_error, T, none)}
|| #{id := Id, status := Status} = T <- Tasks].
- Statuses:
pending,running,paused,completed,failed. pausedwithpaused_reason<<"db_closed">>means the task's local database is not open. It resumes when the database opens.- A one-shot task fails fast; only continuous tasks retry.
Health checks
GET /health answers 200 {"status":"ok"} without authentication. It says the HTTP listener is up; it does not open a database or check a store.
curl -s localhost:8080/health
GET /ready is the readiness check. It also needs no token. It answers 200 {"status":"ready"} when the database and replication processes run, and 503 {"status":"not_ready","failed":[...]} otherwise, where failed names the checks that are down among databases, stores and replication. Point a load balancer or orchestrator at it.
curl -s localhost:8080/ready
For a check on one database, read GET /db/:db with a token. It answers 404 db_not_found when the database does not exist; it does not create it.
curl -s -H "authorization: Bearer $TOKEN" \
localhost:8080/db/kb
barrel_server records no HTTP request metrics and serves no /metrics route yet.