edg#

edg is an expression-based data generator driven by a purpose-built language called edg-lang.

It was built to help you understand and prepare for how your database will behave when production traffic arrives and prevent against logic and performance regressions. Connect to one of the supported databases, create database objects, seed lots of realistic data quickly, and define workloads (transactional or otherwise). Concurrent workers and real-time throughput reporting will take of the rest.

Query arguments are written as expressions compiled at startup, giving you access to global constants, random data generation, reference lookups, and a bunch of random distributions.

Quick start#

Install with the install script.

curl -fsSL https://raw.githubusercontent.com/codingconcepts/edg-releases/main/install.sh | sh

Install via Homebrew.

brew install codingconcepts/tap/edg

Run all of the config steps for the built-in TPC-C workload (one of 16 built-in workloads).

edg workload tpcc all \
  --driver <DRIVER> \
  --url <CONNECTION_STRING> \
  --workers 100 \
  --duration 10m

Run a custom workload.

edg all \
  --driver <DRIVER> \
  --url <CONNECTION_STRING> \
  --config <FILE/HTTP PATH TO CONFIG>.edg \
  --workers 10 \
  --duration 10m

Who is edg for?#

If you interact with databases in any way, edg will have a place in your arsenal. Whether you’re a database administrator, a developer, or a product manager, edg:

• Gives you a reproducible tool for testing workloads; even with complex relationships.
• Can compare the impact of migrations from one database to another.
• Can compare the impact of schema changes on database performance.
• Can reconcile ETL pipelines betweend databases for consistency.
• Can stress test your database, with high concurrency to isolate performance bottlenecks.
• Allows you to run complex workloads and assert the performance and end result (e.g. consistency).

Supported databases#

Supported features#

☑️ Supported · 🚫 Not yet supported · ⚠️ Not applicable

SQLite concurrency#

SQLite is embedded and allows a single writer at a time, so a multi-worker write workload will fail with database is locked (5) (SQLITE_BUSY) unless the database is configured for it. edg passes the connection URL to the driver verbatim and doesn’t rewrite pragmas or clamp the pool size, so tuning is yours to do:

# WAL lets readers run alongside the single writer, busy_timeout makes
# writers wait for the lock instead of failing immediately, and txlock
# starts transactions as writers so they never have to upgrade.
edg run \
  --driver sqlite \
  --url "file:edg.db?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate" \
  --config workload.edg

Rules of thumb:

• journal_mode(WAL) - readers no longer block on the writer. WAL is persisted in the database file, so it only needs setting once.
• busy_timeout(5000) - a writer that finds the lock held retries for up to 5s rather than erroring straight away. Raise it for write-heavy runs.
• _txlock=immediate - needed for any transaction that reads before it writes. A default (deferred) transaction takes a read lock first, and if another writer commits in the meantime the upgrade fails immediately with database is locked (517), which busy_timeout does not retry. Starting as BEGIN IMMEDIATE takes the write lock up front, so the wait happens where busy_timeout applies.
• --pool-size 1 - serialises every statement through one connection, which removes SQLITE_BUSY entirely at the cost of concurrency. Useful when you want a deterministic run rather than a throughput number.
• :memory: gives each connection its own private database, so pair it with --pool-size 1 or use file::memory:?cache=shared.