Percona
Versions
| Run as | Image | Server | DuckDB | Digest |
|---|---|---|---|---|
| Percona | perconalab/ducksdb-mysql-engine:9.7-duckdb-v0.2.4@sha256:222a06f9b717eefded72e21681e2c45f343cafc1b66655ffcf9c90745117a566 | MySQL 9.7.0 + Percona ducksdb-mysql-engine v0.2.4 | 2.0.0 | sha256:222a06f9b717eefded72e21681e2c45f343cafc1b66655ffcf9c90745117a566 |
| Percona (rebuilt on DuckDB 1.5.5) | dbench/percona-duckdb:9.7-v0.2.4-duckdb-1.5.5 | MySQL 9.7.0 + Percona ducksdb-mysql-engine v0.2.4 rebuilt on DuckDB 1.5.5 (not a Percona release) | 1.5.5 | sha256:daf82527ec330d92477177ed4bc13badef772a2c67270d5076c8fa6ace9a0394 |
Observed in this run
Generated the same way for every system from its result files: record counts per measure and status, and the reasons given for anything that is not ok.
Percona
- M1: ok 22
- M2: ok 2
- M3: n/a 1, ok 1
- M4: ok 5
- M5: ok 120
- M7: ok 24
- M8: ok 3
- M3 after_checkpoint (n/a): the Percona DuckDB engine has no checkpoint statement (CHECKPOINT is a syntax error and the engine has no function for it)
Percona (rebuilt on DuckDB 1.5.5)
- M1: ok 22
- M4: ok 5
- M7: ok 24
Settings that differ from the defaults
| Setting | Value | Reason | File |
|---|---|---|---|
skip-log-bin | Binary logging is off on every MySQL-family server (MySQL, Percona and AliSQL enable it by default, MariaDB does not): replication is not under test and the default differs between families. | systems/percona/mysqld-args.j2 | |
innodb_buffer_pool_size | 4G | The InnoDB side only holds the catalog and row-store staging; the data lives in DuckDB. | systems/percona/mysqld-args.j2 |
innodb_parallel_read_threads | 16 | Same as the InnoDB baseline: only COUNT(*) without WHERE and CHECK TABLE use parallel read threads. | systems/percona/mysqld-args.j2 |
secure_file_priv | /data | Server-side LOAD DATA INFILE may read only the read-only dataset mount. | systems/percona/mysqld-args.j2 |
local_infile | OFF | Bulk load is server-side from /data; client-side LOCAL loads stay disabled. | systems/percona/mysqld-args.j2 |
max_connections | 200 | Headroom for the 32-stream throughput run plus probes and cancel connections. | systems/percona/mysqld-args.j2 |
join_buffer_size | 256M | Per-join memory comparable to Postgres work_mem=256MB, same as the InnoDB baseline. | systems/percona/mysqld-args.j2 |
sort_buffer_size | 256M | Per-sort memory comparable to Postgres work_mem=256MB, same as the InnoDB baseline. | systems/percona/mysqld-args.j2 |
tmp_table_size | 4G | Internal temporary tables stay in memory up to mem_gb/16, same as the InnoDB baseline. | systems/percona/mysqld-args.j2 |
max_heap_table_size | 4G | Same cap for MEMORY-engine temporary tables, which use the smaller of the two sizes. | systems/percona/mysqld-args.j2 |
temptable_max_ram | 4G | RAM the TempTable engine may use before spilling, equal to the caps above. | systems/percona/mysqld-args.j2 |
innodb_redo_log_capacity | 8G | A redo log of mem_gb/8 keeps checkpoints from throttling the row-store staging load. | systems/percona/mysqld-args.j2 |
DUCKSDB_MEMORY_LIMIT | 0.75 x mem_gb = 48GB | Caps DuckDB's memory (the engine that runs the queries) at 75% of the container, the same share the InnoDB baseline gives its buffer pool; the rest covers MySQL, connections and the OS. | systems/percona/README.md |
DUCKSDB_TEMP_DIR | /var/lib/mysql/duckdb_tmp | Explicit spill location for DuckDB, on the datadir volume (created at start) instead of the engine default; same spill-location rule for all engines. | systems/percona/README.md |
MYSQL_ROOT_PASSWORD | empty | Passwordless root, as every MySQL-family adapter. | systems/percona/README.md |
README
MySQL 9.7.0 + Percona ducksdb-mysql-engine v0.2.4 (ENGINE=DuckDB)
Image perconalab/ducksdb-mysql-engine:9.7-duckdb-v0.2.4, pinned by digest sha256:222a06f9b717eefded72e21681e2c45f343cafc1b66655ffcf9c90745117a566 (the image reference in dbench/systems/percona.py carries it; the resolved digest is also recorded in every result as versions.image_digest). Track A. The dataset is mounted read-only at /data.
The same engine on DuckDB 1.5.5 (percona-duckdb-1.5.5)
v0.2.4 embeds DuckDB 2.0.0, while MariaDB embeds 1.5.5 and AliSQL a 1.5.2 snapshot, so raw query times mix the engine with its DuckDB version. For a comparison at the same DuckDB version, the benchmark author (who maintains this engine) rebuilt the same engine release on DuckDB 1.5.5. It is not a Percona release: an image built locally on the bench host, dbench/percona-duckdb:9.7-v0.2.4-duckdb-1.5.5, from the engine commit and recipe recorded in its lock entry (build_inputs). The only change from v0.2.4 is a compatibility header (common/duckdb_compat.h) that selects the three DuckDB C++ calls that differ between 1.5 and 2.0; settings, schema and loading are those of percona below. The server reports the embedded DuckDB version and source id, recorded as versions.duckdb and versions.duckdb_source_id, and its overhead (M7) is measured against plain DuckDB 1.5.5. It was added to the talk-sf100 run after the other systems had been measured.
How settings are applied
The image entrypoint starts mysqld --no-defaults ..., so a my.cnf mount is ignored. Every server setting is therefore a mysqld command-line argument, rendered from mysqld-args.j2 with the container limits and passed as the container command. Two engine settings are environment variables. The container runs as root (the image's default); the read-only /data mount causes no chown problem because the entrypoint only creates the datadir.
Non-default settings
At 64 GB and 16 cores (mem_gb = container memory):
| Setting | Value | Reason |
|---|---|---|
env DUCKSDB_MEMORY_LIMIT | 0.75 x mem_gb = 48GB | Caps DuckDB's memory (the engine that runs the queries) at 75% of the container, the same share the InnoDB baseline gives its buffer pool; the rest covers MySQL, connections and the OS. |
innodb_buffer_pool_size | mem_gb/16 = 4G | The InnoDB side only holds the catalog and row-store staging (the load_convert path); the data lives in DuckDB, so a large pool would only take memory from DuckDB. |
innodb_parallel_read_threads | cores in the DB cpuset = 16 | Same as the InnoDB baseline. |
secure_file_priv | /data | Server-side LOAD DATA INFILE may read only the read-only dataset mount. |
local_infile | OFF | Loading is server-side. |
max_connections | 200 | Headroom for the 32-stream throughput run plus probes. |
join_buffer_size, sort_buffer_size | 256M | Same as the InnoDB baseline. |
tmp_table_size, max_heap_table_size, temptable_max_ram | mem_gb/16 = 4G | Same as the InnoDB baseline. |
innodb_redo_log_capacity | mem_gb/8 = 8G | Same as the InnoDB baseline. |
env DUCKSDB_TEMP_DIR | /var/lib/mysql/duckdb_tmp | Explicit spill location for DuckDB, on the datadir volume (created at start) instead of the engine default; same spill-location rule for all engines. |
env MYSQL_ROOT_PASSWORD | empty | Passwordless root, as every MySQL-family adapter. |
All formulas except the buffer pool are shared with the InnoDB baseline (baseline_conf.py). --skip-log-bin: binary logging is off on every MySQL-family server (on by default for MySQL, Percona and AliSQL, off for MariaDB); replication is not under test and the default differs between families. Everything else is the server default.
Disclosed limitations
- DuckDB threads: no setting. The v0.2.4 engine exposes no DuckDB thread-count knob, so DuckDB auto-detects the thread count from the container's CPU set (the 16 pinned cores). Neither a status variable nor a system variable shows the detected count, so it is not observable and is not recorded.
- DuckDB version 2.0.0, read from the server:
versions()records the status variablesDucksdb_duckdb_version(v2.0.0) andDucksdb_duckdb_source_id(d54cbbe3df, the same source id as the official DuckDB 2.0.0 release), andengine: ducksdb-mysql-engine v0.2.4. The plain-DuckDB imagedbench/duckdb:2.0.0(systemduckdb-2.0.0) allows an overhead comparison on the same DuckDB version. (Earlier engine releases exposed no version, and v0.2.3 turned out to be built on an unreleased DuckDB snapshot rather than the stated 1.5.3; that is why the version is now read at runtime.) Ducksdb_pushdown_countis a global counter; see Pushdown.
Schema
create_schema(sf, "engine"): the shared TPC-H DDL with ENGINE=DuckDB and primary keys. The engine accepts PRIMARY KEY declarations (SHOW CREATE TABLE keeps them). No other indexes. kind="rowstore" is plain InnoDB with foreign-key indexes. Each MySQL schema is one DuckDB file, <datadir>/tpch.duckdb. The image creates no database; the adapter creates tpch on first connect.
Loading
load_bulk: engine schema,LOAD DATA INFILEper CSV part (the server's COPY fast path), row count check against the manifest, thenANALYZE TABLEon all tables (supported). Same procedure as every MySQL-family adapter.load_convert: drops and recreates the tables as InnoDB with the engine schema (primary keys, no foreign-key indexes, so the converted tables equal the bulk-loaded ones), loads them withLOAD DATA INFILE(timed asextra.rowstore_load_s), checks row counts, then runsALTER TABLE t ENGINE=DuckDBper table, thenANALYZE TABLEon all tables, asload_bulkdoes, so both paths end in the same state.elapsed_sis the conversion plus analyze, not the staging load.
Pushdown
execute() reads SHOW GLOBAL STATUS LIKE 'Ducksdb_pushdown_count' on the query's own connection before and after the query (outside the timed region) and sets pushed to duckdb if it rose, fallback if not, unknown if the counter could not be read. pushdown(sql) does the same for an arbitrary statement. The counter is server-wide, so the delta is attributable to one query only while a single stream runs (the power run); under concurrent streams it is ambiguous, so the throughput run (M4) records pushed = "unknown" for Percona when streams > 1 (pushdown_counter_scope = "global").
Timeouts on pushed queries
Tested with a heavy pushed query (3-way cross join of a 2M-row DuckDB table, 5 s limit, docker test test_heavy_pushed_query_times_out_and_server_recovers): the server-side max_execution_time stops it (error 3024 after about 5 s) and the next query succeeds.
Measurement notes
Queries run with the session max_execution_time set to the per-query timeout; cancel() is KILL QUERY. storage_bytes() is du -sb of the datadir excluding binlog.* (none are written, binary logging is off), so it counts the .duckdb file and its WAL plus the InnoDB catalog, redo log and tablespaces. Queries use the mechanical MySQL dialect translation shared with the InnoDB baseline.