quiver

Dump, load, copy and stream MySQL and MariaDB from one consistent snapshot, at the speed of the server.

Its own protocol client, its own columnar batches, its own network transport.

Install · Quick start · How it works · Numbers · Across the network

$ quiver dump -h db.example -u backup -B shop --compress zstd -o ./shop
dumped 48211907 rows in 312 chunks into ./shop
8 readers at the start, 8 at most, 16 sessions, 4 renderers, 16 cores to use
Global_read_lock held for 41ms

Documentation of every command and of the masking language: https://jongleb.github.io/quiver/ (quiver docs DIR writes the same site from the binary at hand).

Why quiver

No client library. quiver speaks the MySQL protocol itself and decodes binary rows from the socket buffer straight into Arrow-layout columns. A whole dump allocates 0.05 words per row.	One read, many targets. An agent next to the source sends columnar batches over many TCP streams, compressed once. Collectors write files or load a server and pass everything on to the next one.
Nobody waits alone at the end. Chunks are read in half-second key steps. A reader with nothing left takes the next steps of a chunk another reader is in, so a chunk that turns out far heavier than planned is read by everyone.	Sized by the machine and the server. Without -t, readers follow the fast physical cores and the server's own sizing. Spare sessions wait in the snapshot and join while more readers read more.
The network tunes itself. Each second the agent measures what got through. It adds streams while the link is the bottleneck, keeps them only if throughput grows, and moves zstd between none, fast, 1 and 3.	Broken connections do not end the dump. Every frame is acknowledged. A broken stream hands its frames to the others and is replaced; a broken control connection is replaced while the agent holds its snapshot.
Schema and rows, chosen apart. --data-table takes rows of some tables and the schema of all. --no-data takes the schema alone. --objects none leaves views, triggers, routines and events out.	Checked, not hoped. The compiler checks every thread for shared state (OxCaml modes). Fast code is compared with a simple model by property tests, and every test is shown to fail on deliberately broken code.
Tables built on the client. With --ingest tablespace, quiver writes a table's InnoDB pages itself on every core, and the server imports the finished file. Writing the rows took 5–9× less time than LOAD DATA on one laptop.	JSON numbers stay exact. MySQL reads some of the doubles it prints in JSON as the next double, so a copy through text changes them. quiver finds those numbers and writes them as exact SQL literals.

How it works

• One snapshot. FLUSH TABLES WITH READ LOCK or MariaDB BACKUP STAGE; every session opens its transaction in parallel under the lock, so the lock lasts one round trip. The binlog position and GTID set are taken before it is released.

• Even chunks. Key ranges are bisected by the server's EXPLAIN estimates and small neighbours are merged, so skewed tables split evenly. Data flows from the first second and the DDL of small tables is taken along the way.
• Tables without a key. A table with neither a primary key nor a unique one over NOT NULL columns is read in one pass, cheap for the server. --keyless-read ranges has every reader take a range of another index's first column instead, at the cost of a lookup per row.
• Separate stages. Readers, renderers (SQL, CSV, zstd) and writers run on their own cores, joined by bounded queues. Doubles are printed by zmij, about 13× faster than Grisu3.

A table whose first 1,200 rows hold 1 MB each: every chunk has the same number of rows, one of them carries 1.2 GB. Four readers, four renderers, the same flags in both runs. Each bar is a part of the dump, from the reader's manifest.

Numbers

Apple M5 laptop, MySQL 8.4 in Docker, 4 threads, 3 runs, warm buffer pool, 1.3 GB with narrow and wide rows, integer and string keys and BLOBs. Scripts: bench/run.sh, bench/load.sh, bench/wire.sh.

Agent to collector:

	before	quiver
1 MB/s per flow, 50 ms RTT, streams grown online	35.7 s on 4 streams	12.6 s	2.8× faster
100 ms RTT, mutual TLS, 16 streams opened at once	2.9 s	1.25 s	2.3× faster
Agent CPU, frames written without copies			31 % less

Install

Release binaries run on Linux (x86_64 and arm64, glibc 2.34 or later: RHEL, Rocky and Alma 9, Debian 12, Ubuntu 22.04 and newer) and on macOS 14 or later on Apple silicon. OpenSSL and zstd are built in.

brew install jongleb/tap/quiver                                                  # macOS and Linux
curl -fsSL https://raw.githubusercontent.com/jongleb/quiver/main/install.sh | sh  # any of them, into ~/.local/bin

Debian, Ubuntu (apt)

curl -fsSL https://jongleb.github.io/quiver/quiver.gpg | sudo tee /usr/share/keyrings/quiver.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/quiver.gpg] https://jongleb.github.io/quiver/deb stable main" \
  | sudo tee /etc/apt/sources.list.d/quiver.list
sudo apt update && sudo apt install quiver

Fedora, RHEL, Rocky, Alma (dnf)

sudo curl -fsSLo /etc/yum.repos.d/quiver.repo https://jongleb.github.io/quiver/quiver.repo
sudo dnf install quiver

Arch Linux (AUR), Docker

yay -S quiver-bin
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/work" ghcr.io/jongleb/quiver dump -h db.example -u backup -o ./dump

From source

You need opam, pkg-config and the zstd and OpenSSL libraries. quiver builds with OxCaml, Jane Street's OCaml, on its libraries (Base, Core, ppx_jane):

git clone https://github.com/jongleb/quiver.git && cd quiver
opam switch create quiver-ox 5.2.0+ox \
  --repos ox=git+https://github.com/oxcaml/opam-repository.git,default
opam install --switch=quiver-ox . --deps-only --with-test
opam exec --switch=quiver-ox -- dune build

install.sh reads QUIVER_INSTALL_DIR and QUIVER_VERSION. Every release also carries the archives, .deb and .rpm packages and their SHA256SUMS on GitHub Releases.

Quick start

quiver dump    # MySQL to files: mydumper-compatible SQL or CSV, optional zstd, checksummed manifest
quiver load    # files to MySQL: parallel, streaming, resumes after a crash
quiver copy    # MySQL to MySQL with no files in between
quiver agent   # MySQL to the network, runs next to the source
quiver collect # the network to files or MySQL, runs next to the target, can forward
quiver relay   # a hop that forwards bytes

Dump and load

quiver dump -h db.example -u backup -p secret -o ./dump           # whole server, threads from the machine
quiver dump -B shop -t 8 --compress zstd -o ./shop                 # one database, 8 readers
quiver dump -T shop.orders --format csv --csv-dialect mysql-load-data -o ./csv
quiver dump -B shop --data-table shop.orders -o ./shop             # every table's schema, rows of orders only
quiver dump -B shop --no-data -o ./schema                          # schema and objects, no rows
quiver dump -T shop.orders --objects none -o ./orders              # without views, triggers, routines, events
quiver dump --snapshot backup-stage -o ./dump                      # MariaDB: block commits only

quiver load -d ./shop -B shop_copy -t 8                            # or: myloader -d ./shop

The layout is mydumper's, so myloader restores a quiver dump too.

Copy server to server

quiver copy -h old.example -B shop --to-host new.example --to-database shop

Each chunk streams from its reading session into a loading session of its own through LOAD DATA LOCAL INFILE, in one transaction, while other chunks are still being read. Foreign keys are added once all data is in.

Building tablespaces

quiver copy -B shop --to-database shop_copy --ingest tablespace --binlog skip

LOAD DATA makes the target insert every row into its B-tree. Sessions that load the same table wait on that table's index latch. With --ingest tablespace, quiver builds the clustered index itself.

1. The target creates the table under a name of quiver's, with its primary key and the secondary keys quiver can build.
2. FLUSH TABLES … FOR EXPORT gives quiver the server's own empty tablespace and .cfg file as a template.
3. Renderers turn batches into InnoDB records of every index. Runs of each table fill clustered pages on threads of their own, and sort their secondary records into a scratch file. When the table's last part closes, each secondary index is merged by key ranges on several threads.
4. Before the server sees the file, quiver checks it the way IMPORT and the server read it.
5. IMPORT TABLESPACE takes the file. quiver checks the row count and any warning, renames the table and has the server add the secondary keys it did not build.

• Types. Every type a dump reads is stored, including JSON in the server's binary form and columns too long for their page (LOB pages). Text in utf8mb4, utf8mb3 and latin1 is stored, and zero dates are kept with --zero-dates keep. STORED generated columns hold the source's own values: quiver reads them for these tables only, and never writes them back through LOAD DATA.
• Keys. Any primary key works, also text in a collation or a float. Rows of such a key are stored in the order the source sorted them, and the target checks that order in its own collation when it imports the table: a table out of order is refused, not stored. A table without a primary key works too; its hidden row ids are numbered by quiver.
• Secondary keys. quiver builds keys over integers, dates and times, decimals, binary strings (prefixes too) and text in a _bin collation into the file, also for a table without a primary key, and checks UNIQUE keys itself, since IMPORT does not. Keys over text in another collation, floats or DESC columns, SPATIAL and FULLTEXT keys are added by the server after the import, in their places, so SHOW CREATE TABLE matches the source. This is the default; --tablespace-keys server has the server add them all after the import.
• Checks. Each stored field and every secondary index record is compared byte for byte with records the server wrote itself.
• Fallback. A table the writer cannot build goes through LOAD DATA, and quiver says why. Partitions and other character sets are such cases. At the end, quiver sums up rows and bytes by way and by reason, and times each table's import and secondary keys.
• What it saves. Writing rows and building byte-ordered keys moves to the client's cores; the import then has more pages to take. Tables that fall back load as before.
• Settings. --tablespace-keys quiver|server, --sort-dir (where sorted secondary records wait; beside the file by default), --build-threads (threads finishing and checking built files, shared by all tables). --read-order largest-tables reads the largest tables first, so their check, import and keys run while the rest is read, and --declare-tables all-at-once prepares every table before the rows; both default to how 1.3.2 worked (largest-chunks, before-rows).
• Requirements. MySQL 8.4 or Percona Server 8.4, and the target's data directory on the same machine (--datadir if quiver sees it under another path).
• Binlog. --binlog skip is required: an import writes no rows to the binlog, so replicas would silently miss them.

quiver collect takes --ingest tablespace as well, so an agent can feed a collector next to the target; the collector tells the agent to read STORED generated columns too, and so does a collector in a chain when a node after it builds tablespaces.

JSON numbers

MySQL prints a double in JSON as its shortest exact text. When it reads JSON text back, its parser takes about one in nine of those texts as the neighbouring double, without a warning. Every tool that moves JSON as text changes such numbers. In a live copy through LOAD DATA, 63 of 553 documents came out different.

quiver checks every JSON value with a model of the server's parser and writes values that would change differently:

• SQL dumps. The value becomes JSON_SET(<text>, <path>, <exact double literal>, …). The server reads an SQL double literal exactly.
• copy to MySQL. Such rows bypass LOAD DATA and are inserted that way in the same transaction.
• CSV dumps. Such a value fails with an error; a CSV file has no way to keep it.
• MariaDB targets. Their JSON is text, so nothing changes there.

Masking

Personal data can be masked as it is read, so the real values never leave the reader: not in dump files, not on the wire, not in the target. dump, copy and agent take the same rules.

quiver copy -h prod.example -B shop --to-host dev.example --to-database shop \
  --mask 'shop.users.name=shape' \
  --mask 'shop.*.*email=email' \
  --mask 'shop.users.phone=shape' \
  --mask 'shop.users.passport=hash' \
  --mask 'shop.users.note=null' \
  --mask 'shop.users.status=text:new' \
  --mask "shop.users.born=sql:MAKEDATE(YEAR(born), 1)" \
  --mask-secret "$MASK_SECRET"

A rule is DATABASE.TABLE.COLUMN=ACTION. Names take * and ?, or backquotes for a name as written; a column takes the first rule naming it.

Action	What the column gets
null	NULL
text:VALUE	this value: text, binary, an ENUM member or SET members
shape	each letter and digit replaced by a keyed one of the same kind (Latin, Cyrillic, digit), the rest kept: +7 (912) 345-67-89 → +3 (481) 905-12-77, Иван Smith → Жуэр Qmpwa
email	shape before the last @, the domain kept
hash	a keyed digest of the same length in hex digits or bytes, or any integer of the column's type
sql:EXPRESSION	the expression, evaluated by the source for each row and cast to the column's type
expr:EXPRESSION	an expression of quiver's masking language (below), evaluated here
keep	the values as read: the column counts as decided for --mask-strict

• Keyed. shape, email and hash are HMAC-SHA256 under --mask-secret: the same value masks the same way in every column and every run, so joins and foreign keys between masked columns still hold. Without a secret, a random key of this run is used.
• Never longer. shape, email and hash keep the number of characters and never add bytes, so a value still fits its column. NULL stays NULL but under null and sql:.
• Refused, not guessed. A rule naming no column of the dump, a column of the key the table is read by, null for a NOT NULL column, an action that gives no value of the column's type or a text: outside an ENUM are errors before anything is read. So is a table whose STORED generated columns a tablespace copy would take from the source. A sql: expression that gives NULL for a NOT NULL column, or makes the source warn (a value cast with loss), fails the copy.
• Recorded. The run prints every masked column, and quiver.json lists them.
• Your own statements. copy, collect and load take --after-load 'DB.TABLE=STATEMENT': the target runs it once every table's rows and keys are in, before foreign keys, views and triggers, in the table's database, with {table} for its name there. Rows pass through the target unmasked before such an UPDATE, so it suits what --mask cannot express.
• As a library. Mask.custom takes any OCaml function of text or binary values.

Your own algorithm

When the actions above are not enough, write the masking policy in a small language of S-expressions. quiver types it against every column before anything is read, compiles it, and runs it on the render threads. It always ends: nothing loops, and a function calls only those defined above it.

; policy.scm
(words names "names.txt")                 ; one fake first name per line, beside the script

(define (phone p)                         ; +7 (912) 345-67-89 → +7 (937) 482-67-89
  (concat "+7 (" (str (keyed-int p 900 999)) ") " (digits p 3) "-" (right p 5)))
(define (person n)                        ; Иван Петров → Дарья Kknyayyubae
  (concat (pick-word n names) " " (title (translit (shape (after n " "))))))

(strict "shop.users")                     ; every column of users decided, or nothing is read
(keep "shop.users.id")
(keep (where (table "shop.users") (column "created_*")))
(mask "shop.users.name"  (person value))
(mask "shop.users.phone" (phone value))
(mask "shop.users.born"                   ; same year, another day
  (keyed-date (str (row id)) (make-date (year value) 1 1) (make-date (year value) 12 31)))
(mask "shop.users.salary" (round-to (keyed-noise value 20) 1000))
(mask "shop.users.card"   (let ((d (digits value 15))) (concat d (luhn d))))
(mask "shop.users.note"   (redact value 3 0))
(mask (where (table "shop.*") (column "*email*") (type text)) (email value))
(mask (where (table "shop.users") (type binary)) (hash-bytes value))

(after-load "shop.orders" "UPDATE {table} SET comment = NULL")

quiver copy … --mask-script policy.scm --mask-secret "$MASK_SECRET"
quiver mask --mask-script policy.scm '(phone value)' '+7 (912) 345-67-89'   # try it, no server

• Selectors. "DB.TABLE.COLUMN" with * and ?, or (where (table …) (column …) (type …)) to take columns by name and by type: text, binary, json, integer, decimal, float, date, datetime, timestamp, time, year, enum, set, geometry. A column takes the first rule naming it, from --mask first, then the scripts in order.
• Decisions. (keep …) copies columns as they are. (strict "DB.TABLE" …) (or --mask-strict) requires every column of those tables to be masked or kept: a column added to the schema later stops the copy with its name, before anything is read, instead of leaking.
• Values. value is the column's value; (row COLUMN) another column of the row as read, before any masking. Text, binary, JSON, ENUM, SET and DECIMAL columns are text, integer columns are integers, DATE a date, DATETIME and TIMESTAMP a date with a time.
• Functions. Text: concat, length, substring, left, right, upper, lower, title, trim, reverse, replace, before, after, before-last, after-last, split-part, index-of, pad-left, pad-right, repeat, format, translit, only-digits, redact, str, int, starts-with, ends-with, contains, like. Numbers: + - * / mod abs min max round-to floor-to clamp. Dates: year month day make-date add-days days-between date->text text->date date-of seconds-of-day make-datetime add-seconds datetime->text text->datetime. Logic: = <> < <= > >= and or not is-null coalesce, if, cond, let, define.
• Keyed by --mask-secret: shape, email, hash, hash-bytes, hash-int, keyed-int, keyed-date, keyed-noise, keyed-uuid, digits, pick, pick-word. The same seed gives the same result in every column and run, so masked keys still join. luhn gives a card number's check digit.
• Safe. NULL goes through functions as in SQL. A type error (as file:line:column), a NULL for a NOT NULL column, or a time finer than its column stops the run. quiver mask --help has the whole reference.

The same rules fit a quiverfile: a map under an option gives it once per leaf, keys joined by dots.

copy:
  mask-script: [policy.scm]
  mask-strict: [shop.orders]
  mask:
    shop.users:
      inn: "expr:(hash-int value)"
      comment: "null"
    shop.orders:
      user_email: email
      id: keep
  after-load:
    shop.orders:
      - "UPDATE {table} SET total = ROUND(total, -2)"

Connecting

quiver dump -o ./dump                                              # no -h: the local socket, found as mysql finds it
quiver dump -S /var/run/mysqld/mysqld.sock -o ./dump               # a given socket (skip_networking)
quiver dump -h 127.0.0.1 -o ./dump                                 # local server over TCP

• Without -h quiver finds the socket as the mysql client does: socket in the [client] groups of my.cnf with !include and !includedir, then MYSQL_UNIX_PORT, then the packaged paths.
• Logins: caching_sha2_password, mysql_native_password, sha256_password with RSA over a socket, MariaDB unix_socket.
• TLS against system roots by default, your CA with --tls-ca, or a pinned certificate with --tls-fingerprint. --connect-timeout bounds connecting, --net-timeout bounds silence.

Settings in a quiverfile

# quiverfile.yaml
socket: /var/run/mysqld/mysqld.sock     # every command
to-socket: /var/run/mysqld/mysqld.sock

copy:                                   # quiver copy alone
  database: shop
  to-database: shop_copy
  threads: 8
  ingest: tablespace
  binlog: skip
  data-table: [shop.orders, shop.items]

Any option can live in a YAML file, under its long name. Options at the top hold for every command; those under a command's name hold for it alone, and a list gives the option once per item. quiver reads --quiverfile FILE, else the file in QUIVERFILE, else quiverfile.yaml in the current directory, and says which. The command line wins over the environment, and both win over the file: each option also reads QUIVER_<NAME> (QUIVER_THREADS, QUIVER_TO_HOST, …). An unknown key is an error, as is a key whose meaning differs between commands (database) at the top. See quiverfile.example.yaml.

Across the network

# next to the target
quiver collect --listen 0.0.0.0:7300 --to-host db.local --to-database shop \
  --wire-cert collector.crt --wire-key collector.key --wire-peers nodes-ca.crt

# next to the source
quiver agent -h old.example -B shop --collector collector.example:7300 \
  --wire-cert agent.crt --wire-key agent.key --wire-peers nodes-ca.crt

Every connection speaks TLS 1.3 and both ends prove who they are: the agent checks that the collector's certificate chains to --wire-peers and names the host it dialled, and the collector accepts only agents whose certificates chain there too. Pinning works as well: give each side the other's self-signed certificate as --wire-peers. On a private network, --wire-tls disabled opts out.

Add hops and targets with the same pieces ($WIRE stands for the three --wire- options):

quiver relay   --listen 0.0.0.0:7300 --forward far.example:7300 $WIRE                  # a hop
quiver collect --listen 0.0.0.0:7300 -o ./shop --forward replica.example:7300 $WIRE  # keep a copy, pass it on

One read of the source reaches a whole chain of targets, each in its own format: files here, a live server there. A chunk counts once every target has committed it.

The wire protocol

One control connection carries the session; S data connections carry frames of batches, every frame with its own two checksums.

A CA for your nodes in three commands

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 3650 \
  -subj /CN=nodes-ca -keyout nodes-ca.key -out nodes-ca.crt
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -subj /CN=collector \
  -keyout collector.key -out collector.csr
openssl x509 -req -in collector.csr -CA nodes-ca.crt -CAkey nodes-ca.key -days 825 \
  -extfile <(echo subjectAltName=DNS:collector.example) -out collector.crt

Repeat the last two for the agent and every relay.

Correctness

• If the requested consistency cannot be given, quiver refuses to start.
• Floats in their shortest exact form, decimals and fractional seconds as stored. A lossy conversion is an error. Zero dates are an error unless kept explicitly.
• JSON doubles the server would read back as their neighbour are written as exact SQL literals.
• A masking rule that names no column, or cannot mask a column without breaking its key or type, stops the run before it reads anything.
• LOAD DATA LOCAL must store exactly the rows sent and raise no warning. Otherwise the part rolls back and the error carries the server's warnings.
• A built tablespace passes quiver's own check of every page, list and LOB before the server sees it. After the import, its row count must match the rows sent.
• Chunks cover each key domain exactly, also when a chunk is divided while it is read: the open edges are first bounded by MIN and MAX inside the snapshot.
• xxh3 of every file in the manifest, checked by quiver load. On the wire every frame and header has its own checksum, inside mutual TLS 1.3 with a key update every 64 GiB.
• Any failure exits non-zero with a typed error and writes no manifest, so an unfinished dump never looks finished. quiver load records each file in the same transaction as its data and resumes after a crash.
• Integration tests on MySQL 8.0 and 8.4 and MariaDB 10.11, 11.4 and 11.8: random tables of every type dumped and restored, Sakila with all objects through myloader and quiver, bank transfers under concurrent writes, resets, silent peers, killed sessions and a full disk.
• Network sessions are property tests: random topologies with relays and chains run while connections are cut, corrupted and reset. Either every node commits every part or every node fails.

Status

quiver is young and moving fast.

Area
Dump, load and copy; separate schema and rows	shipped
Tablespaces built on the client (MySQL 8.4, Percona 8.4)	new in 1.3
Masking as rows are read; statements after the load	new in 1.4
Unix sockets, my.cnf, every common login	shipped
Agent, collector, relays and chains, mutual TLS 1.3	shipped
Online tuning of streams, compression, readers; chunks divided while read	shipped
Resuming a network transfer from the same snapshot	next
Postgres and other providers	after MySQL is done

Run quiver <command> --help for every option.

License

quiver is free software under the GNU Affero General Public License v3.0 or any later version. You may use, change and share it. If you distribute a modified quiver, or run one as a service that others reach over a network, you must offer them its source under the same licence. The vendored zmij (vendor/zmij, MIT), xxHash (vendor/xxhash, BSD-2-Clause) and YAMLx (vendor/yamlx, AGPL-3.0-only) keep their own licences. Release binaries also carry OpenSSL (Apache-2.0) and zstd (BSD-3-Clause), whose licences ship next to the binary in licenses/.