Cluster#

The cluster command group provides a distributed workload runner using CockroachDB for coordination. Multiple agents across pods, nodes, clouds, or regions execute workloads with synchronized stage changes and global QPS control.

Architecture#

The cluster uses three components:

• Coordinator - HTTP server that manages jobs, orchestrates stages, and distributes QPS and seed counts across agents
• Agents - Workers that register with the coordinator, receive commands via CockroachDB changefeeds, and execute workloads against a target database
• Coordination database - A CockroachDB instance that stores job history, agent liveness (via row-level TTL), and commands (via sinkless changefeeds)

graph TD
    CLI["   edg cluster submit   "]
    Coord["&nbsp;&nbsp; Coordinator &nbsp;&nbsp;<br/>&nbsp;&nbsp; edg cluster coordinator &nbsp;&nbsp;"]
    CRDB[("&nbsp;&nbsp; CockroachDB &nbsp;&nbsp;<br/>&nbsp;&nbsp; coordination DB &nbsp;&nbsp;")]
    A1["&nbsp;&nbsp; Agent 1 &nbsp;&nbsp;<br/>&nbsp;&nbsp; edg cluster agent &nbsp;&nbsp;<br/>&nbsp;&nbsp; us-east &nbsp;&nbsp;"]
    AN["&nbsp;&nbsp; Agent N &nbsp;&nbsp;<br/>&nbsp;&nbsp; edg cluster agent &nbsp;&nbsp;<br/>&nbsp;&nbsp; eu-west &nbsp;&nbsp;"]
    Target[("&nbsp;&nbsp; Target &nbsp;&nbsp;<br/>&nbsp;&nbsp; database &nbsp;&nbsp;")]

    CLI -- "&nbsp; POST /cluster/jobs &nbsp;" --> Coord
    Coord -- "&nbsp; write jobs & commands &nbsp;" --> CRDB
    CRDB -- "&nbsp; changefeed<br/>&nbsp; commands &nbsp;" --> A1
    CRDB -- "&nbsp; changefeed<br/>&nbsp; commands &nbsp;" --> AN
    A1 -- "&nbsp; heartbeat &<br/>&nbsp; register &nbsp;" --> CRDB
    AN -- "&nbsp; heartbeat &<br/>&nbsp; register &nbsp;" --> CRDB
    A1 -- "&nbsp; execute workload &nbsp;" --> Target
    AN -- "&nbsp; execute workload &nbsp;" --> Target

Subcommands#

Coordinator#

Start the coordinator, pointing it at a CockroachDB instance for coordination:

edg cluster coordinator \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable"

The coordinator creates the coordination tables on first startup and listens on port 3001 by default.

Token authentication#

Pass --token to require all endpoints to include a matching Edg-Cluster-Token header:

edg cluster coordinator \
  --crdb-url "postgres://root@localhost:26257?sslmode=disable" \
  --token my-secret-token

Coordinator flags#

Coordinator endpoints#

Agent#

Start an agent, pointing it at the same CockroachDB instance:

edg cluster agent \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable" \
  --region us-east

The agent registers itself, heartbeats every 10 seconds, and watches for commands via a CockroachDB sinkless changefeed. If the agent crashes or stops heartbeating, it is automatically reaped by row-level TTL within 30 seconds.

Agent flags#

Submit#

Submit a workload to the cluster:

edg cluster submit \
  --coordinator-url http://localhost:3001 \
  --target-url "postgres://root@localhost:26257?sslmode=disable" \
  --config workload.edg \
  --duration 10m \
  --workers 10 \
  --regions us-east,eu-west \
  --stream

Submit flags#

Separating seed from run#

By default each agent runs whichever of up and seed the config defines before its first stage. --sections narrows that:

# Point the fleet at an estate that already exists.
edg cluster submit --config workload.edg --target-url "$URL" --sections run -d 10m

This matters more in cluster mode than on a single process, because otherwise every agent re-checks the same schema before any of them can start.

A cluster submission must include run. Agents build their runner when the first stage command arrives, so a prepare-only job would dispatch nothing and complete having built nothing; the coordinator rejects it with a 400 rather than accept a submission that does nothing. Seed with edg jobs submit --sections up,seed or locally with edg seed.

deseed and down are accepted but have never been run by cluster agents, so naming them changes nothing.

Per-job environment#

env('X') and env_nil('X') read the OS environment of whichever process evaluates the config. In cluster mode that process is the agent, whose environment is fixed at deploy time - so without --env, every coalesce(env_nil('...'), default) in a submitted config falls back to its default across the whole fleet.

--env attaches key/value pairs to the submission itself. The coordinator stores them with the job and the agent resolves env()/env_nil() against them, falling back to its own OS environment for anything not set. That makes one config file parameterisable per submission:

edg cluster submit --config workload.edg --target-url "$URL" \
  -e SHAPE_PEAK_HOUR=3 -e SHAPE_AMP=0.9

edg cluster submit --config workload.edg --target-url "$URL" \
  --env-file ./peak.env

SHAPE_PEAK_HOUR and SHAPE_AMP mean nothing to edg - workload.edg reads them with env_nil(). See Shapes for a config that does.

--env is repeatable rather than comma-separated, so a value containing a comma survives intact. --env-file is a deliberately dumb KEY=VALUE per line format: # comments and blank lines are skipped, and a value is the rest of the line verbatim - no quote stripping, no escapes, no variable expansion. An explicit --env beats --env-file, and a job’s env beats the agent’s OS environment.

Use this for parameters, not secrets. Job env is stored in the coordination database in plaintext and is readable by anyone who can read cluster_jobs.

Job env does not change the agent’s process environment. It only affects env()/env_nil() while that job’s config is evaluated, so a submitted job cannot repoint the agent’s own coordination database or license key, and the next job the agent runs sees only its own env.

Status#

edg cluster status --coordinator-url http://localhost:3001

edg cluster status <job-id> --coordinator-url http://localhost:3001

Status flags#

Stream#

edg cluster stream <job-id> --coordinator-url http://localhost:3001

Stream flags#

Cancel#

edg cluster cancel <job-id> --coordinator-url http://localhost:3001

Returns 409 Conflict if the job is not running.

Cancel flags#

Agents List#

edg cluster agents --coordinator-url http://localhost:3001

Agents flags#

Reset#

Drop and recreate all coordination tables. Useful for clearing stale state or starting fresh:

edg cluster reset \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable"

Reset flags#

Global QPS Distribution#

When a job specifies QPS limits (via stages in the config), the coordinator divides the target QPS evenly across active agents:

per_agent_qps = ceil(total_qps / active_agent_count)

The coordinator monitors agent counts every 5 seconds. When an agent joins or leaves, QPS is automatically redistributed via a changefeed command.

Agent Liveness#

Agents heartbeat every 10 seconds by updating their last_seen timestamp. CockroachDB row-level TTL automatically deletes agents that haven’t heartbeated in 30 seconds. When an agent is reaped, the coordinator detects the count change and redistributes QPS to the remaining agents.

Per-Region Metrics#

When running in cluster mode, all Prometheus metrics include a region label matching the agent’s --region flag. This enables per-region dashboards and alerts:

rate(edg_query_duration_seconds_sum{region="us-east"}[1m])
  / rate(edg_query_duration_seconds_count{region="us-east"}[1m])

Persistent Job History#

Unlike the jobs server (which stores history in memory), cluster jobs are persisted in CockroachDB. Job history survives coordinator restarts.

Quick Start#

# 1. Start CockroachDB
cockroach start-single-node --insecure --listen-addr=localhost:26257

# 2. Create the coordination database
cockroach sql --insecure -e "CREATE DATABASE edg_cluster"

# 3. Start the coordinator
edg cluster coordinator \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable"

# 4. Start agents (in separate terminals)
edg cluster agent \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable" \
  --region us-east

edg cluster agent \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable" \
  --region eu-west

# 5. Submit a workload
edg cluster submit \
  --target-url "postgres://root@localhost:26257?sslmode=disable" \
  --config workload.edg \
  --duration 5m \
  --workers 10 \
  --stream