Configuration

Barca needs no configuration file. Settings come from command-line flags, environment variables and an optional barca.toml, in that order:

CLI flag > environment variable > barca.toml > built-in default

For example, three ways to select the staging environment:

CLI flag

barca get pipeline.py --env staging

Environment variable

export BARCA_ENV=staging
barca get pipeline.py

barca.toml

barca.toml

default_env = "staging"

Where things live

Barca keeps its cache and run history in .barca/, in the project root. Results are always written there first. A remote store, when configured, holds a copy of the result files and backups of the metadata database.

Path	What it is	Safe to delete
.barca/metadata.db	Run history and the record of every cached result (a SQLite file).	Only to discard local history: every asset then recomputes on the next run (with a store, barca state restore puts a backup back instead). Delete metadata.db-wal and metadata.db-shm with it, with no barca command running.
.barca/metadata.db-wal, .barca/metadata.db-shm	SQLite’s write-ahead log for metadata.db.	Not on their own: they belong to metadata.db and can hold rows not yet in it.
.barca/metadata.db.lock	A lock file held briefly while a barca process reads or writes the database.	Yes, when no barca command is running.
.barca/artifacts/<node>/<run_hash>.<ext>	One file per cached result: .json, .pkl or .parquet. A partitioned asset has one directory per key (pipeline.py--sales_region_emea/).	Yes. A result whose file is gone is computed again when a run needs to read it, or downloaded again if it is in a remote store. Prefer --refresh <asset> to force a recompute.
.barca/envs/<name>/	The same metadata.db and artifacts/ for a named environment (--env <name>).	As above, per environment.
.barca/backups/<name>.db, <name>.json, <name>.unsent	With a store ([remote] uri): the five newest backups of metadata.db, each with its sidecar, and a marker on one the store has not confirmed yet. Under .barca/envs/<name>/backups/ for a named environment.	Yes; then only the store’s copies are left to restore from.
.barca/metadata.db.pre-restore	The database barca state restore replaced. barca never reads it.	Yes.
.barca/serve.lock	Held by a running barca serve, so barca state restore refuses to run under it.	Yes, when no server is running.
.barca/metadata.db.base, .prev, .pull-*, .push-*	Left by barca 0.25 and earlier, which shared history through the store. Not used.	Yes.
.barca/run-owners/<token>.fifo	From 0.20.0: a marker (a named pipe, no data) that a barca process holds open while it has a run in flight. A killed run is recognised by its marker having nobody at the other end. The process removes it when its runs have ended; one left by a killed process is removed by a later run an hour after no running run names it.	When barca history shows no running run. While one does, its marker is what can show that it was killed.
.barca/sql-cache/	With remote storage: copies of remote results that barca sql downloaded, laid out by URI (sql-cache/s3/<bucket>/...), each with a .version file. The first thing the cache size limit evicts, least recently used first.	Yes, always.
.barca/range-cache/<run_hash>-<version>/	With remote storage: the byte ranges lazy inputs (duckdb.DuckDBPyRelation, pl.LazyFrame) read from a store object, one file per range, reused by later runs on this machine. Evicted with the leftovers, least recently used first.	Yes, always; the ranges are fetched again when read.
.barca/staging/<pid>/	Temporary files of a running worker writing to a remote @sink (a remote dataset sink stages a directory per write) or fetching a remote input. Directories of dead workers are removed when a worker starts.	Yes, when no barca command is running.
.barca/gc-state.json	From 0.25: what the last full measurement of .barca/ found, so the check after each command does not walk the artifact directories (Cache size).	Yes; the next check measures again.
.barca/.gitignore	Written by barca, containing *, so git ignores everything under .barca/.	It is written again.

Some rows appear only in the situations named. For backups see barca docs remote; for a missing artifact see barca docs cache, “A cached result whose artifact is missing”.

Do not edit files under .barca/, and do not open metadata.db with another program while barca is running. To look at results use barca sql and barca status.

Project root

The nearest barca.toml at or above the directory you run barca from marks the project root. Barca changes into the root before doing anything, so barca.toml, .barca/, node ids and the working directory of every step are the same wherever in the project you run it. File arguments are read relative to where you typed them. Without a barca.toml above you, the current directory is the root.

An empty barca.toml is enough to mark a root. When the root is not the current directory, barca prints barca: project root: <path> on stderr.

barca.toml

Every key is optional.

# The environment used when neither --env nor BARCA_ENV is given.
default_env = "dev"

[discovery]
# Which files a command with no file arguments reads. See the Discovery page.
exclude = ["scratch/**", "notebooks/**"]   # skipped in addition to the built-in list
include = ["pipelines/**/*.py"]            # only these files (replaces the walk)

[cache]
# How much disk .barca/ may use. "20GB", "500MB", "20GiB", a number of bytes, "10%" of the
# filesystem, or "off". Default: 10% of the filesystem. See "Cache size" below.
max_size = "20GB"

[server]
# Seconds `barca serve` waits for runs in flight on SIGTERM or SIGINT before putting them back
# in its run queue for the next start.
drain_timeout = 30
# How many runs `barca serve` executes at once; the others wait in its run queue (experimental).
max_runs = 4
# The most the weight= of the steps of every run of `barca serve` may add up to (experimental).
# Default: none (each run has only its own capacity).
capacity = 4
# Seconds a run under `barca serve` may take before it is stopped. Default: 0, no limit.
run_timeout_seconds = 3600

[execution]
# The most the weight= of the steps running at once may add up to (experimental).
# Default: the pool size. See "Heavy steps" on the Decorators page.
capacity = 4
# How many worker processes a run starts at most. Default: the cores, or the capacity when
# it is smaller. BARCA_POOL_SIZE wins over it.
pool_size = 4

[remote]
# Root of the remote store. Setting it turns remote storage on:
# artifacts at {uri}/{env}/artifacts, backups of the history at {uri}/{env}/state/backups/.
# A plain directory (a shared mount) also works.
uri = "s3://my-bucket/barca/my-project"

# Optional overrides, used as written (the environment name is not added):
# artifacts_uri = "s3://my-bucket/custom-artifacts"

transfer_concurrency = 4    # artifact uploads and downloads in flight at once (at least 1)
transfer_timeout = 600      # seconds one transfer attempt may run (at least 1)

# Passed to fsspec.filesystem(protocol, ...), one table per protocol: s3, gcs, abfs.
[remote.storage_options.s3]
endpoint_url = "https://<account-id>.r2.cloudflarestorage.com"

[server]
# The barca serve that is the team's ledger (experimental). get and run still run here,
# read cache hits from its ledger and post what they compute (needs [remote] with the
# server's store); status, history and stats answer from it; --server runs a command there.
url = "https://barca.internal.example.com"

Key	Default	Meaning
default_env	default	Environment name when --env and BARCA_ENV are absent.
require_env	false	true: a command that reads or writes an environment exits 2 unless --env or BARCA_ENV names it, even with default_env set. See Environments.
[discovery].include	none	Glob patterns; only matching files are discovered. See Discovery.
[discovery].exclude	none	Glob patterns skipped in addition to the built-in list.
[cache].max_size	10% of the filesystem	Size limit for .barca/; past it barca evicts down to 80% of it. "off" for no limit. See Cache size.
[execution].capacity	the pool size	Experimental. The most the weight= of the steps running at once may add up to; a whole number of at least 1. A step heavier than it runs alone. See Heavy steps.
[server].drain_timeout	30	Seconds barca serve waits for runs in flight once stopped with SIGTERM or SIGINT, before it puts them back at the head of its run queue (they start again after the restart) and exits 0. 0 requeues at once; a second signal cancels them instead. --drain-timeout overrides it.
[execution].pool_size	the cores, or the capacity when smaller	How many worker processes a run starts at most; a whole number of at least 1. Without it a run starts no more workers than its capacity (every step weighs at least 1). BARCA_POOL_SIZE overrides it. barca plan (pool_source) and barca doctor say which applied.
[server].capacity	none	Experimental. The most the weight= of the steps of every run of barca serve may add up to, admitted first come, first served across runs (a heavy step is not overtaken by lighter ones). Bounds each run’s pool too. --capacity overrides it. See Scheduling.
[server].run_timeout_seconds	0 (no limit)	Seconds a run under barca serve may take, from when it leaves the queue; a run that takes longer is stopped and recorded as an error naming the setting. --run-timeout overrides it; Schedule(..., timeout_seconds=N) wins for its ticks. Before 0.31.0 the limit was a fixed 600 s.
[server].max_runs	one per core	Experimental. How many runs barca serve executes at once; the others wait in its run queue, in order. At least 1. --max-runs overrides it.
[remote].uri	none	Root of the remote store.
[remote].artifacts_uri	{uri}/{env}/artifacts	Artifact location, used as written.
[remote].transfer_concurrency	4	Transfers in flight at once.
[remote].transfer_timeout	600	Seconds one transfer attempt may run.
[remote.storage_options.<protocol>]	none	Options for the fsspec filesystem of that protocol.
[server].url	none	The barca serve that is the team’s ledger (BARCA_SERVER overrides it): http:// or https://, with any proxy path prefix. get and run run here and read and post through its ledger (requires [remote] pointing at the server’s store); status, history and stats answer from it; sql resolves its views from it; a bare --server runs the command there; --local ignores it. See Server mode and Server Mode.

An unknown key or a malformed file is a usage error (exit 2) that names the key and lists the valid ones:

invalid ./barca.toml: TOML parse error at line 1, column 1
unknown field `defualt_env`, expected one of `default_env`, `require_env`, `remote`, `discovery`, `cache`, `server`, `execution`

Keep credentials out of barca.toml; they come from the environment.

Cache size

From 0.25, .barca/ has a size limit: [cache].max_size, or BARCA_CACHE_MAX_SIZE, and with neither, 10% of the filesystem .barca/ is on. "off" removes the limit (earlier versions had none: every new run hash added a file and kept the old one).

• KB, MB, GB, TB are decimal (20GB is 20 × 10⁹ bytes); KiB, MiB, GiB, TiB are binary; a bare number is bytes. A percentage is of the filesystem’s size, read at each check.
• The size is everything under .barca/, every environment, the metadata databases included (they are counted, never deleted).

After every barca get and barca run, after every run under barca serve, and once a minute under barca serve, barca checks the size. Over the limit, it deletes files in this order until .barca/ is at or below 80% of the limit:

1. Leftovers: staging directories of processes that are no longer running, *.moved-aside* directories, artifact files no metadata row names (what barca state reset and barca state restore leave behind), sql-cache/ copies and range-cache/ directories (least recently used first).
2. Superseded results: a result whose node (or partition key) has a newer successful result. Oldest first.
3. Local copies the artifact store holds: downloaded again when needed. Least recently read first.
4. Current results: computed again when needed. Least recently read first.

It never deletes what the command produced, read or returned; results a run in flight holds; anything but leftovers and superseded results while another run is in progress on this .barca/; the current result of a freshness=Manual node the store does not hold; an unnamed artifact file while a run is in progress or in the minute after it was written; the metadata databases, their backups and the copies reset and restore moved aside; or anything outside .barca/. The remote store is never touched: bound it with your bucket’s lifecycle rules.

When something was evicted, stderr says so in one line, and --json results gain cache: {size_bytes, limit_bytes, evicted_files, evicted_bytes}. An evicted result that a later run needs is downloaded or computed again under the same run hash, with reason evicted. barca status shows it as stale with reason evicted. barca gc runs the same eviction on demand (--dry-run to see it first, --max-size for a one-off limit; when it cannot reach its target, held says what holds the rest), and barca doctor reports the size against the limit and the free space. The full description is in barca docs cache, “Disk use”.

[remote].state, [remote].state_uri and [remote].push_retries were removed in 0.26.0 with the shared history (RFC-0009). A file that still sets one is a usage error (exit 2) naming it, even under BARCA_REMOTE=off: remove the key. Backups need no setting.

Environment variables

Variables that override barca.toml:

Variable	Overrides	Notes
BARCA_ENV	default_env	Overridden by --env.
BARCA_REQUIRE_ENV	require_env	1 (or true, yes) requires --env or BARCA_ENV; 0 (or false, no) does not; anything else exits 2.
BARCA_CAPACITY	[execution].capacity	Experimental. A whole number of at least 1; anything else exits 2.
BARCA_SERVER_RECONNECT	none	Experimental. Seconds a --server client following a run (and logs -f --server) waits for a restarting server to come back (default 60; 0 gives up at once). Not a number of seconds exits 2.
BARCA_CACHE_MAX_SIZE	[cache].max_size	Same values; an invalid one exits 2. For a container that sets its limit when it is deployed.
BARCA_REMOTE	process override	off uses local artifacts/history, takes no backups and overrides all remote settings. Unset/empty keeps normal resolution; other values exit 2.
BARCA_REMOTE_URI	[remote].uri	Enough on its own to turn remote storage and history backups on.
BARCA_TRANSFER_CONCURRENCY	[remote].transfer_concurrency	A positive integer. In 0.18.0 an invalid value exits 3, not 2.
BARCA_TRANSFER_TIMEOUT	[remote].transfer_timeout	Positive integer seconds, per attempt. In 0.18.0 an invalid value exits 3, not 2.
BARCA_STORAGE_OPTIONS	[remote.storage_options.*]	JSON keyed by protocol, merged over the toml tables key by key.
FSSPEC_<PROTOCOL>_<OPTION>	one storage option	fsspec’s own convention (FSSPEC_S3_ENDPOINT_URL, FSSPEC_GCS_PROJECT). Overridden by both of the above.
BARCA_SERVER	[server].url	The server a bare --server uses. Overridden by --server <url>.
BARCA_ARTIFACT_URI	the artifact location	Used as written, with no environment name added, and history stays local. Kept from 0.4.0. Warns when combined with a non-default environment. No backups are taken. Prefer BARCA_REMOTE_URI.

An empty value counts as unset. BARCA_REMOTE=off overrides the URI, transfer and storage-options settings above while preserving the selected local environment. BARCA_STATE, BARCA_STATE_URI and BARCA_PUSH_RETRIES were removed in 0.26.0: setting one is a usage error (exit 2) naming it.

Variables with no barca.toml key:

Variable	Effect
BARCA_OUTPUT	json or pretty: the output format when no flag is given. Any other value exits 2.
BARCA_PROGRESS_SECS	Seconds between still running lines on stderr (default 15; 0 turns them off).
BARCA_POOL_SIZE	Number of Python worker processes (default: [execution].pool_size, else the available cores, or the capacity when it is smaller).
BARCA_TRACE_TIMING	When set, prints a timing breakdown on stderr.
BARCA_TELEMETRY	datadog sends finished runs to Datadog. See Telemetry for it and the DD_* variables.
BARCA_COMM_COST_SECONDS	The scheduler’s estimate of the cost of dispatching one step.

The CLI contract marks each variable stable or experimental. BARCA_SOCKET, BARCA_WORKER and BARCA_WORKER_ID are set by barca for its own worker processes; do not set them.

Credentials are read by the storage libraries from their usual variables (AWS_*, GOOGLE_APPLICATION_CREDENTIALS, AZURE_STORAGE_*). BARCA_REMOTE_URI plus those is a complete remote setup with no barca.toml. See Remote Storage.

Server mode

A configured server is the team’s ledger (experimental; barca docs server-mode, Deploying, RFC-0007). Which server:

--server <url> > BARCA_SERVER > [server].url in barca.toml

What a command does with it:

Command	Without a flag	--server [<url>]	--local
get, run	runs here; cache hits from the server’s ledger, results posted to it	runs on the server, printed here	runs here against this machine’s ledger; nothing sent
status, history, stats	answered by the server (status adds local per node)	answered by the server	this machine’s records
list	this machine’s files	the deployed nodes	this machine’s files
sql	views and barca.* tables from the server’s ledger; the query runs here	the same, from that server	this machine’s results

barca get total                                                # here, against the ledger
barca get total --server https://barca.internal.example.com   # that server, for this command
export BARCA_SERVER=https://barca.internal.example.com
barca get total --server                                       # the configured server
barca get total --local                                        # this machine only

A bare --server with no server configured is a usage error (exit 2). When the URL came from BARCA_SERVER or barca.toml, barca prints [barca] server <url> (<source>) once on stderr. --local is a usage error with --server. A local get/run against the ledger needs [remote] pointing at the server’s store (a usage error otherwise, naming both stores); a server that cannot be reached is exit 3 naming it, with --local as the remediation. --env with --server must name the server’s environment.

Environments (--env)

--env <name> keeps a separate cache, history and remote location per name, so dev and prod never share results. get, run, status, sql, history, stats and serve accept it. plan and list read no state and do not. The name comes from --env, then BARCA_ENV, then default_env, then default, and may contain letters, digits, ., _ and -; anything else exits 2.

get and run print the environment and artifact store they use once at startup, on stderr, when the environment comes from default_env or an artifact store is configured, so a one-off command on a machine set up for another environment says which one it uses:

[barca] env prod (default_env in barca.toml); artifact store local (.barca/envs/prod/artifacts)
[barca] env dev (--env); artifact store s3://bucket/proj/dev/artifacts

The source in brackets is --env, BARCA_ENV, default_env in barca.toml or built-in default; credentials and signed URI queries are redacted. The line is never on stdout, so --json output is unchanged. barca serve always prints it, with the metadata DB and its format added: ...; metadata .barca/envs/prod/metadata.db (db format 1) (see Deploying).

On a machine with more than one environment (.barca/metadata.db for default, and each directory of .barca/envs/), a command that names none with --env or BARCA_ENV prints one note on stderr naming the environment it uses and the ones there are, in every output mode (stdout and --json are unchanged):

[barca] note: no --env given, so this uses env staging (default_env in barca.toml); this project has 2 environments (prod, staging). Pass --env NAME or set BARCA_ENV; require_env = true in barca.toml (or BARCA_REQUIRE_ENV=1) makes it required

require_env = true at the top level of barca.toml (or BARCA_REQUIRE_ENV=1, which beats it; BARCA_REQUIRE_ENV=0 turns it off) makes the environment required: every command that reads or writes one (get, run, status, history, logs, stats, sql, gc, state and serve) exits 2 without --env or BARCA_ENV, naming the environments there are, even when default_env is set, and barca doctor reports it as a failed environment check. plan, list and docs read no environment and are not affected.

	env = default	named env <e>
local history	.barca/metadata.db	.barca/envs/<e>/metadata.db
local artifacts	.barca/artifacts/	.barca/envs/<e>/artifacts/
remote artifacts	{uri}/default/artifacts/	{uri}/<e>/artifacts/
remote history	{uri}/default/state/metadata.db	{uri}/<e>/state/metadata.db

barca get total --env dev
barca status total --env dev     # what is cached in dev
barca sql "select * from total" --env dev

An environment that has never been used has no results: barca status --env <e> reports every node as never_run, and barca sql "show tables" --env <e> lists nothing. Neither is an error, so a misspelled name is not reported.