DuckDB inside OLTP databases

Percona

Versions

Run asImageServerDuckDBDigest
Perconaperconalab/ducksdb-mysql-engine:9.7-duckdb-v0.2.4@sha256:222a06f9b717eefded72e21681e2c45f343cafc1b66655ffcf9c90745117a566MySQL 9.7.0 + Percona ducksdb-mysql-engine v0.2.42.0.0sha256:222a06f9b717eefded72e21681e2c45f343cafc1b66655ffcf9c90745117a566
Percona (rebuilt on DuckDB 1.5.5)dbench/percona-duckdb:9.7-v0.2.4-duckdb-1.5.5MySQL 9.7.0 + Percona ducksdb-mysql-engine v0.2.4 rebuilt on DuckDB 1.5.5 (not a Percona release)1.5.5sha256:daf82527ec330d92477177ed4bc13badef772a2c67270d5076c8fa6ace9a0394

Observed in this run

Generated the same way for every system from its result files: record counts per measure and status, and the reasons given for anything that is not ok.

Percona

Percona (rebuilt on DuckDB 1.5.5)

Settings that differ from the defaults

SettingValueReasonFile
skip-log-binBinary logging is off on every MySQL-family server (MySQL, Percona and AliSQL enable it by default, MariaDB does not): replication is not under test and the default differs between families.systems/percona/mysqld-args.j2
innodb_buffer_pool_size4GThe InnoDB side only holds the catalog and row-store staging; the data lives in DuckDB.systems/percona/mysqld-args.j2
innodb_parallel_read_threads16Same as the InnoDB baseline: only COUNT(*) without WHERE and CHECK TABLE use parallel read threads.systems/percona/mysqld-args.j2
secure_file_priv/dataServer-side LOAD DATA INFILE may read only the read-only dataset mount.systems/percona/mysqld-args.j2
local_infileOFFBulk load is server-side from /data; client-side LOCAL loads stay disabled.systems/percona/mysqld-args.j2
max_connections200Headroom for the 32-stream throughput run plus probes and cancel connections.systems/percona/mysqld-args.j2
join_buffer_size256MPer-join memory comparable to Postgres work_mem=256MB, same as the InnoDB baseline.systems/percona/mysqld-args.j2
sort_buffer_size256MPer-sort memory comparable to Postgres work_mem=256MB, same as the InnoDB baseline.systems/percona/mysqld-args.j2
tmp_table_size4GInternal temporary tables stay in memory up to mem_gb/16, same as the InnoDB baseline.systems/percona/mysqld-args.j2
max_heap_table_size4GSame cap for MEMORY-engine temporary tables, which use the smaller of the two sizes.systems/percona/mysqld-args.j2
temptable_max_ram4GRAM the TempTable engine may use before spilling, equal to the caps above.systems/percona/mysqld-args.j2
innodb_redo_log_capacity8GA redo log of mem_gb/8 keeps checkpoints from throttling the row-store staging load.systems/percona/mysqld-args.j2
DUCKSDB_MEMORY_LIMIT0.75 x mem_gb = 48GBCaps DuckDB's memory (the engine that runs the queries) at 75% of the container, the same share the InnoDB baseline gives its buffer pool; the rest covers MySQL, connections and the OS.systems/percona/README.md
DUCKSDB_TEMP_DIR/var/lib/mysql/duckdb_tmpExplicit spill location for DuckDB, on the datadir volume (created at start) instead of the engine default; same spill-location rule for all engines.systems/percona/README.md
MYSQL_ROOT_PASSWORDemptyPasswordless root, as every MySQL-family adapter.systems/percona/README.md

README

MySQL 9.7.0 + Percona ducksdb-mysql-engine v0.2.4 (ENGINE=DuckDB)

Image perconalab/ducksdb-mysql-engine:9.7-duckdb-v0.2.4, pinned by digest sha256:222a06f9b717eefded72e21681e2c45f343cafc1b66655ffcf9c90745117a566 (the image reference in dbench/systems/percona.py carries it; the resolved digest is also recorded in every result as versions.image_digest). Track A. The dataset is mounted read-only at /data.

The same engine on DuckDB 1.5.5 (percona-duckdb-1.5.5)

v0.2.4 embeds DuckDB 2.0.0, while MariaDB embeds 1.5.5 and AliSQL a 1.5.2 snapshot, so raw query times mix the engine with its DuckDB version. For a comparison at the same DuckDB version, the benchmark author (who maintains this engine) rebuilt the same engine release on DuckDB 1.5.5. It is not a Percona release: an image built locally on the bench host, dbench/percona-duckdb:9.7-v0.2.4-duckdb-1.5.5, from the engine commit and recipe recorded in its lock entry (build_inputs). The only change from v0.2.4 is a compatibility header (common/duckdb_compat.h) that selects the three DuckDB C++ calls that differ between 1.5 and 2.0; settings, schema and loading are those of percona below. The server reports the embedded DuckDB version and source id, recorded as versions.duckdb and versions.duckdb_source_id, and its overhead (M7) is measured against plain DuckDB 1.5.5. It was added to the talk-sf100 run after the other systems had been measured.

How settings are applied

The image entrypoint starts mysqld --no-defaults ..., so a my.cnf mount is ignored. Every server setting is therefore a mysqld command-line argument, rendered from mysqld-args.j2 with the container limits and passed as the container command. Two engine settings are environment variables. The container runs as root (the image's default); the read-only /data mount causes no chown problem because the entrypoint only creates the datadir.

Non-default settings

At 64 GB and 16 cores (mem_gb = container memory):

SettingValueReason
env DUCKSDB_MEMORY_LIMIT0.75 x mem_gb = 48GBCaps DuckDB's memory (the engine that runs the queries) at 75% of the container, the same share the InnoDB baseline gives its buffer pool; the rest covers MySQL, connections and the OS.
innodb_buffer_pool_sizemem_gb/16 = 4GThe InnoDB side only holds the catalog and row-store staging (the load_convert path); the data lives in DuckDB, so a large pool would only take memory from DuckDB.
innodb_parallel_read_threadscores in the DB cpuset = 16Same as the InnoDB baseline.
secure_file_priv/dataServer-side LOAD DATA INFILE may read only the read-only dataset mount.
local_infileOFFLoading is server-side.
max_connections200Headroom for the 32-stream throughput run plus probes.
join_buffer_size, sort_buffer_size256MSame as the InnoDB baseline.
tmp_table_size, max_heap_table_size, temptable_max_rammem_gb/16 = 4GSame as the InnoDB baseline.
innodb_redo_log_capacitymem_gb/8 = 8GSame as the InnoDB baseline.
env DUCKSDB_TEMP_DIR/var/lib/mysql/duckdb_tmpExplicit spill location for DuckDB, on the datadir volume (created at start) instead of the engine default; same spill-location rule for all engines.
env MYSQL_ROOT_PASSWORDemptyPasswordless root, as every MySQL-family adapter.

All formulas except the buffer pool are shared with the InnoDB baseline (baseline_conf.py). --skip-log-bin: binary logging is off on every MySQL-family server (on by default for MySQL, Percona and AliSQL, off for MariaDB); replication is not under test and the default differs between families. Everything else is the server default.

Disclosed limitations

Schema

create_schema(sf, "engine"): the shared TPC-H DDL with ENGINE=DuckDB and primary keys. The engine accepts PRIMARY KEY declarations (SHOW CREATE TABLE keeps them). No other indexes. kind="rowstore" is plain InnoDB with foreign-key indexes. Each MySQL schema is one DuckDB file, <datadir>/tpch.duckdb. The image creates no database; the adapter creates tpch on first connect.

Loading

Pushdown

execute() reads SHOW GLOBAL STATUS LIKE 'Ducksdb_pushdown_count' on the query's own connection before and after the query (outside the timed region) and sets pushed to duckdb if it rose, fallback if not, unknown if the counter could not be read. pushdown(sql) does the same for an arbitrary statement. The counter is server-wide, so the delta is attributable to one query only while a single stream runs (the power run); under concurrent streams it is ambiguous, so the throughput run (M4) records pushed = "unknown" for Percona when streams > 1 (pushdown_counter_scope = "global").

Timeouts on pushed queries

Tested with a heavy pushed query (3-way cross join of a 2M-row DuckDB table, 5 s limit, docker test test_heavy_pushed_query_times_out_and_server_recovers): the server-side max_execution_time stops it (error 3024 after about 5 s) and the next query succeeds.

Measurement notes

Queries run with the session max_execution_time set to the per-query timeout; cancel() is KILL QUERY. storage_bytes() is du -sb of the datadir excluding binlog.* (none are written, binary logging is off), so it counts the .duckdb file and its WAL plus the InnoDB catalog, redo log and tablespaces. Queries use the mechanical MySQL dialect translation shared with the InnoDB baseline.