Cluster PRO#

The cluster command group provides a distributed workload runner using CockroachDB for coordination. Multiple agents across pods, nodes, clouds, or regions execute workloads with synchronized stage changes and global QPS control.

Architecture#

The cluster uses three components:

• Coordinator - HTTP server that manages jobs, orchestrates stages, and distributes QPS and seed counts across agents
• Agents - Workers that register with the coordinator, receive commands via CockroachDB changefeeds, and execute workloads against a target database
• Coordination database - A CockroachDB instance that stores job history, agent liveness (via row-level TTL), and commands (via sinkless changefeeds)

graph TD
    CLI["   edg cluster submit   "]
    Coord["&nbsp;&nbsp; Coordinator &nbsp;&nbsp;<br/>&nbsp;&nbsp; edg cluster coordinator &nbsp;&nbsp;"]
    CRDB[("&nbsp;&nbsp; CockroachDB &nbsp;&nbsp;<br/>&nbsp;&nbsp; coordination DB &nbsp;&nbsp;")]
    A1["&nbsp;&nbsp; Agent 1 &nbsp;&nbsp;<br/>&nbsp;&nbsp; edg cluster agent &nbsp;&nbsp;<br/>&nbsp;&nbsp; us-east &nbsp;&nbsp;"]
    AN["&nbsp;&nbsp; Agent N &nbsp;&nbsp;<br/>&nbsp;&nbsp; edg cluster agent &nbsp;&nbsp;<br/>&nbsp;&nbsp; eu-west &nbsp;&nbsp;"]
    Target[("&nbsp;&nbsp; Target &nbsp;&nbsp;<br/>&nbsp;&nbsp; database &nbsp;&nbsp;")]

    CLI -- "&nbsp; POST /cluster/jobs &nbsp;" --> Coord
    Coord -- "&nbsp; write jobs & commands &nbsp;" --> CRDB
    CRDB -- "&nbsp; changefeed<br/>&nbsp; commands &nbsp;" --> A1
    CRDB -- "&nbsp; changefeed<br/>&nbsp; commands &nbsp;" --> AN
    A1 -- "&nbsp; heartbeat &<br/>&nbsp; register &nbsp;" --> CRDB
    AN -- "&nbsp; heartbeat &<br/>&nbsp; register &nbsp;" --> CRDB
    A1 -- "&nbsp; execute workload &nbsp;" --> Target
    AN -- "&nbsp; execute workload &nbsp;" --> Target

Subcommands#

Coordinator#

Start the coordinator, pointing it at a CockroachDB instance for coordination:

edg cluster coordinator \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable"

The coordinator creates the coordination tables on first startup and listens on port 3001 by default.

Token authentication#

Pass --token to require all endpoints to include a matching Edg-Cluster-Token header:

edg cluster coordinator \
  --crdb-url "postgres://root@localhost:26257?sslmode=disable" \
  --token my-secret-token

Coordinator flags#

Coordinator endpoints#

Agent#

Start an agent, pointing it at the same CockroachDB instance:

edg cluster agent \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable" \
  --region us-east

The agent registers itself, heartbeats every 10 seconds, and watches for commands via a CockroachDB sinkless changefeed. If the agent crashes or stops heartbeating, it is automatically reaped by row-level TTL within 30 seconds.

Agent flags#

Submit#

Submit a workload to the cluster:

edg cluster submit \
  --coordinator-url http://localhost:3001 \
  --target-url "postgres://root@localhost:26257?sslmode=disable" \
  --config workload.edg \
  --duration 10m \
  --workers 10 \
  --regions us-east,eu-west \
  --stream

Submit flags#

Status#

edg cluster status --coordinator-url http://localhost:3001

edg cluster status <job-id> --coordinator-url http://localhost:3001

Status flags#

Stream#

edg cluster stream <job-id> --coordinator-url http://localhost:3001

Stream flags#

Cancel#

edg cluster cancel <job-id> --coordinator-url http://localhost:3001

Returns 409 Conflict if the job is not running.

Cancel flags#

Agents List#

edg cluster agents --coordinator-url http://localhost:3001

Agents flags#

Reset#

Drop and recreate all coordination tables. Useful for clearing stale state or starting fresh:

edg cluster reset \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable"

Reset flags#

Global QPS Distribution#

When a job specifies QPS limits (via stages in the config), the coordinator divides the target QPS evenly across active agents:

per_agent_qps = ceil(total_qps / active_agent_count)

The coordinator monitors agent counts every 5 seconds. When an agent joins or leaves, QPS is automatically redistributed via a changefeed command.

Agent Liveness#

Agents heartbeat every 10 seconds by updating their last_seen timestamp. CockroachDB row-level TTL automatically deletes agents that haven’t heartbeated in 30 seconds. When an agent is reaped, the coordinator detects the count change and redistributes QPS to the remaining agents.

Per-Region Metrics#

When running in cluster mode, all Prometheus metrics include a region label matching the agent’s --region flag. This enables per-region dashboards and alerts:

rate(edg_query_duration_seconds_sum{region="us-east"}[1m])
  / rate(edg_query_duration_seconds_count{region="us-east"}[1m])

Persistent Job History#

Unlike the jobs server (which stores history in memory), cluster jobs are persisted in CockroachDB. Job history survives coordinator restarts.

Quick Start#

# 1. Start CockroachDB
cockroach start-single-node --insecure --listen-addr=localhost:26257

# 2. Create the coordination database
cockroach sql --insecure -e "CREATE DATABASE edg_cluster"

# 3. Start the coordinator
edg cluster coordinator \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable"

# 4. Start agents (in separate terminals)
edg cluster agent \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable" \
  --region us-east

edg cluster agent \
  --crdb-url "postgres://root@localhost:26257/edg_cluster?sslmode=disable" \
  --region eu-west

# 5. Submit a workload
edg cluster submit \
  --target-url "postgres://root@localhost:26257?sslmode=disable" \
  --config workload.edg \
  --duration 5m \
  --workers 10 \
  --stream