Wirekite Benchmarks

Oracle to Google AlloyDB PostgreSQL - 1B rows

Wirekite — Mon, 11 Aug 2025 19:45:39 GMT

Google AlloyDB is a cluster-packaged cloud-hosted PostgreSQL-compatible database engine that uses Google’s cloud storage for its storage world. A more in-depth overview of AlloyDB can be found here.

For our benchmark, we used a single-node “cluster”, with the node’s compute instance specified as a Google N2-Standard-64 host. We used the same type of compute instance - N2-Standard-64 - on the Oracle extraction side.

After adding support for “local” file loading to the Wirekite PostgreSQL data loader - this change was needed to load any PostgreSQL managed-cloud target - we were able to use it without additional changes to load to AlloyDB.

Benchmark Details

This is a pure “how fast can Wirekite load AlloyDB” benchmark. We used our 1 billion row “firenibble” table, extracted from Oracle and loaded to AlloyDB, as the basis for this benchmark.

Table

We used our standard firenibble table, with the Oracle table definition shown below.

The table has 16 columns, one of them is a numeric primary key, few columns are numbers of various shapes and sizes, few strings and few dates. This makes the table pretty simple to be absorbed by most databases.

The table also has 1 billion rows.

CREATE TABLE app_db.firenibble (
    f0    number(38)    NOT NULL,
    f1    number(38),
    f2    number(38),
    f3    int,
    f4    float,
    f5    float,
    f6    float,
    f7    float,
    f8    varchar2(128),
    f9    varchar2(128),
    f10    date,
    f11    date,
    f12    date,
    f13    varchar2(128),
    f14    varchar2(128),
    f15    varchar2(128),
    PRIMARY KEY (f0)
);

The 1 billion rows.

SQL> select count(*) from app_db.firenibble;

  COUNT(*)
----------
1000000000

1 row selected.

SQL>

Oracle

Our Oracle environment is a GCP N2-standard-64 host in the us-central1 region. The host OS is Linux RHEL8 (Redhat) and Oracle is Oracle version 19c.

Benchmark Run

For AlloyDB, we created a single AlloyDB node configured as an N2-Standard-64 host. The cluster and node are in the GCP us-central1 region - same as our Oracle world - to minimize network lag.

We ran with 64 extraction threads.

One observation is in test runs against AlloyDB, we reached 100% CPU utilization, but at 64 threads, we only reached 75% CPU utilization. We believe we saturated the network between our Oracle compute instance and the AlloyDB node at about 48 threads.

And this was the result. It took 24 minutes 35 secs to move 1 billion rows from Oracle to AlloyDB. The Oracle extract alone took 21 min 25 secs, so the load to AlloyDB lagged about 2 min 50 secs behind the extract from Oracle. The transfer rate works out to 677,966 rows per second.

Here’s the confirmation queries run on AlloyDB:

wirekite=> select now();
             now
------------------------------
 2025-08-10 20:10:29.26418+00
(1 row)

wirekite=> select sum(mark::int) from wirekite_progress where mark::int <> 0;
    sum
------------
 1000000000
(1 row)

wirekite=> select count(*) from firenibble;
   count
------------
 1000000000
(1 row)

PostgreSQL to Snowflake - 1B rows - 89x Faster

Wirekite — Sun, 08 Jun 2025 07:00:00 GMT

So we did our first benchmark and we are thrilled to share the results. We chose to compare our benchmark with Airbyte since that is one of the most recognized data migration enterprise software out there.

This was inspired from a benchmark that Airbyte published - https://airbyte.com/blog/postgres-replication-performance-benchmark-airbyte-vs-fivetran.

Benchmark Details

Table

As you can see we used the same table that Airbyte used. It has a numeric primary key column, some number columns, some string columns and some date columns. This fits nicely with most databases.

CREATE TABLE firenibble
(
    f0 BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    f1 BIGINT,
    f2 BIGINT,
    f3 INTEGER,
    f4 DOUBLE PRECISION,
    f5 DOUBLE PRECISION,
    f6 DOUBLE PRECISION,
    f7 DOUBLE PRECISION,
    f8 VARCHAR COLLATE pg_catalog."default",
    f9 VARCHAR COLLATE pg_catalog."default",
    f10 DATE,
    f11 DATE,
    f12 DATE,
    f13 VARCHAR COLLATE pg_catalog."default",
    f14 VARCHAR COLLATE pg_catalog."default",
    f15 VARCHAR COLLATE pg_catalog."default"
) tablespace benchmark ;

And here’s the 1 billion rows. The query to do a count takes a very long time.

wirekite=# select count(*) from firenibble;
   count
------------
 1000000000
(1 row)

Machine Configurations

For Postgres we created an n2-standard-64 GCP instance with all the standard configurations - 64 CPU’s and 256 GB RAM in the us-central region.

For Snowflake we created a Large size warehouse with Standard Type also in the us-central1 region to minimize the network distance.

Airbyte Run

We created the firenibble table with various sizes to get the sense of how airbyte functions. We did Full Refresh Overwrite and made sure the table did not exist in the target.

We tried a bunch of runs. Some failed for various reasons. It is frustrating when we saw the run failed at 975 million rows.

But then it succeeded !!

1 billion rows migrated in 12 hours 6 minutes and 29 seconds.

Execution logs and details as reported by Airbyte.

2025-03-08 19:21:20 info sync summary: {
  "status" : "completed",
  "recordsSynced" : 1000000000,
  "bytesSynced" : 439710415164,
  "startTime" : 1741418118670,
  "endTime" : 1741461680232,
  "totalStats" : {
    "bytesCommitted" : 439710415164,
    "bytesEmitted" : 439710415164,
    "destinationStateMessagesEmitted" : 100000,
    "destinationWriteEndTime" : 1741461680072,
    "destinationWriteStartTime" : 1741418118737,
    "meanSecondsBeforeSourceStateMessageEmitted" : 3,
    "maxSecondsBeforeSourceStateMessageEmitted" : 100000,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 68,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 48,
    "recordsEmitted" : 1000000000,
    "recordsCommitted" : 1000000000,
    "recordsFilteredOut" : 0,
    "bytesFilteredOut" : 0,
    "replicationEndTime" : 1741461680087,
    "replicationStartTime" : 1741418118670,
    "sourceReadEndTime" : 1741460800423,
    "sourceReadStartTime" : 1741418118738,
    "sourceStateMessagesEmitted" : 100000
  }

Wirekite Run

Same Postgres instance. Same Snowflake instance. Same table. Same data.

1 billion rows were migrated in 8 minutes 9 seconds.

# cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Fri Mar 14 03:54:04 UTC 2025
FINISH: Fri Mar 14 04:02:13 UTC 2025
ELAPSED: 0 hrs 8 min 9 sec

And here’s the table in Snowflake.

Oracle to Snowflake - 10 million ops (inserts + updates + deletes) - 6 mins 7 secs

Wirekite — Sun, 08 Jun 2025 07:00:00 GMT

So we set up a Change Data Capture replication pipeline from Oracle to Snowflake and wanted to benchmark how it performs under a “typical” OLTP load. This article discusses the various characteristics of the benchmark

Benchmark Details

The most important decision here was to determine how to do a replication benchmark, specifically what constitutes a typical load profile of an OLTP database. We realized that a typical OLTP database has the following characteristics

It has multiple tables.
It has some inserts.
It has somewhat more updates.
It has some deletes.
It has small transactions with lot of commits, mostly single row operations and commits.

Given the above characteristics we decided on doing the following for the benchmark.

We will have 10 tables.
We will have 2 million single row inserts.
We will have 6 million single row updates.
We will have 2 million single row deletes.
We will have auto commit on, causing all the above operations to have corresponding commit.

Table Structure

We will have 10 tables called firenibble 1 to 10.

CREATE TABLE app_db.firenibble1 (
  f0 number(38) NOT NULL,
  f1 number(38) ,
  f2 number(38) ,
  f3 int ,
  f4 float ,
  f5 float ,
  f6 float ,
  f7 float ,
  f8 varchar2(128) ,
  f9 varchar2(128) ,
  f10 date ,
  f11 date ,
  f12 date ,
  f13 varchar2(128) ,
  f14 varchar2(128) ,
  f15 varchar2(128) ,
  primary key (f0)
);

Insert

Insert will essentially be new primary key insert.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Update

Update will be updating of two columns on a primary key value.

update firenibble1 set f2 = 280396830734761092, f9 = 'n6ChQjN9dfxIl6nc6SNgIzSahfeXfAef' where f0 = 19998;

Delete

Delete will be essentially a primary key delete.

delete from firenibble1 where f0 = 19998;

DML Operations Sequence

So the exact sequence will look like this. I know we could have been more granular mixing the inserts, updates and deletes even more, and as we do more benchmarks we might try that strategy but for starters this looked pretty good. Also we wanted to see how our replication performed with large number of insert only, update only and delete only loads. Also note that there is no relationships between the load and the archive logs, in the sense there is no alter system switch logfile happening anywhere causing the inserts, updates and deletes to cross binary log boundaries.

1 million inserts
3 million updates
1 million deletes
1 million inserts
3 million updates
1 million deletes.

We will also makes sure that all inserts, updates and deletes happen on 10 tables in a sequence so that the operations are granularily staggered over all tables and not contiguous on one table and the next. Which means, the inserts look like this (and so does the updates and deletes).

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble2 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble3 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble4 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble5 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble6 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble7 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble8 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble9 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble10 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Benchmark Sequence

A choice we had was to make was do one of the following

Have the wirekite replication running. Run the load and wait for the last event of the load to appear on the target. Measure the time between when the load started and the last event appeared.
Run the load and note down the binlog number and position at which the load started and wait for the load to finish. Now start the wirekite replication from the starting binlog number and position and wait for the last event of the load to appear on the target. Measure the time between when the wirekite replication started and the last event appeared on the target.

We decided to go with option 2. The primary reason being that in option 1 wirekite replication was waiting for the events to finish successfully on the database, and essentially we were not only benchmarking wirekite, but also the database, with the boundaries between the two completely blurred. So we preferred the pre-loaded database with the changes, so that we are only benchmarking wirekite and nothing else.

Machine Configuration

For Oracle we created an n2-standard-16 GCP instance with all the standard configurations - 8 CPU’s and 64 GB RAM in the us-central region. We installed Ubuntu version 24 and Oracle version 19. Our archive log size was 330 MB.

For Snowflake we created a Large Size Warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

Oracle Run

We wanted to mention this a bit to talk about the uphill challenge that was getting oracle to be up and running and importing data into it.

First it was the installation which we thought we could do using the command line. But once we started playing with response files we realized it was almost impossible to get the right response file. There are hundreds of options that go in oracle response file which makes it impossible to find the right combination. We kept on running into errors.

So we decided to use the GUI. Getting X11 forwarding working took some time (that DISPLAY variable) but we were able to do it. But then Oracle GUI would show up with buttons missing from it - https://forums.oracle.com/ords/apexds/post/19c-runinstaller-shows-blank-screen-for-15-minutes-9680.

We realized that some given environment variable has to be set to make sure the GUI runs as intended. Once we did that we got the right GUI and got Oracle installed.

And then came the challenge of configuring the database with sensible values so that you can actually load a reasonable amount of data in it. We had to configure

Configure and start the LISTENER. Understand LISTENER.ORA and TNSNAMES.ORA.
Figure out the differences between PFILE and SPFILE and the workings of SCOPE.
Turn off Automatic Shared Memory Management - MEMORY_TARGET = 0
Turn off Automatic Management - SGA_TARGET = 0
Set SGA_MAX_SIZE
Set DB_CACHE_SIZE
Set SHARED_POOL_SIZE
Set LARGE_POOL_SIZE
Modify a few kernel parameters (SHMMAX, SHMALL, file open limits), modify sysctl.conf and limits.conf and reboot the instance a few times.
Create a separate TABLESPACE to hold the table with the right number and size of data files and make sure they extend automatically.
Grant privileges on this TABLESPACE to the right USER.
Create the table with the right TABLESPACE for data but also the right TABLESPACE for INDEX otherwise it will create index in the default tablespace.
Figure out SQL LOADER to load the data from a csv into the table. Make sure you have the right commit frequency.
Monitor DBA_FREE_SPACE and add data files when it comes close.
Play with V$SESSION and V$SESSION_WAIT to see what is happening.
Play with EXPLAIN PLANS since Oracle gives a thousand ways to see the actual explain plan.

No Airbyte Run

We could not do an airbyte run since it requires an enterprise level connector. So this is just our benchmark.

Wirekite Run

So how long did it take for wirekite to move 10 million changes — 6 minutes and 7 seconds.

cat wirekite_cdc.log | grep -E 'START|FINISH|ELAPSED'
START: Mon Apr 28 02:47:15 UTC 2025
FINISH: Mon Apr 28 02:53:22 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 6 min 7 sec

PostgreSQL to Firebolt - 1B rows - 188x Faster

Wirekite — Sat, 07 Jun 2025 07:00:00 GMT

So we did our first Firebolt benchmark. And wow this was quite a ride. Firebolt is very different from other warehouses - the tradeoffs are really interesting. Some good, Some bad, in our opinion. As I go through the benchmark I will talk about that.

We chose to compare our benchmark with Airbyte since that is one of the most recognized data migration enterprise software out there.

This was inspired from a benchmark that Airbyte published - https://airbyte.com/blog/postgres-replication-performance-benchmark-airbyte-vs-fivetran.

I do want to mention here that Firebolt benchmark is a little different, and slightly slower than what it could possibly be and the reason being that our database server is in GCP while our Firebolt bucket is in AWS. That is definitely a network hop, that will do some latency, as opposed to going from GCP to GCP. This probably explains why Firebolt benchmarks are slightly slower.

Benchmark Details

Table

As usual we decided to use our same firenibble table that Airbyte used in their benchmark. We have standardized on this table because we want our benchmarks to be almost a fair comparison between various databases.

The table has a numeric primary key column, some number columns, some string columns and some date columns. This fits nicely with most databases.

CREATE TABLE firenibble
(
    f0 BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    f1 BIGINT,
    f2 BIGINT,
    f3 INTEGER,
    f4 DOUBLE PRECISION,
    f5 DOUBLE PRECISION,
    f6 DOUBLE PRECISION,
    f7 DOUBLE PRECISION,
    f8 VARCHAR COLLATE pg_catalog."default",
    f9 VARCHAR COLLATE pg_catalog."default",
    f10 DATE,
    f11 DATE,
    f12 DATE,
    f13 VARCHAR COLLATE pg_catalog."default",
    f14 VARCHAR COLLATE pg_catalog."default",
    f15 VARCHAR COLLATE pg_catalog."default"
) tablespace benchmark ;

And here’s the 1 billion rows.

wirekite=# select count(*) from firenibble;
   count
------------
 1000000000
(1 row)

Machine Configurations

For Postgres we created an n2-standard-64 GCP instance with all the standard configurations - 64 CPU’s and 256 GB RAM in the us-central region.

For Firebolt we created an engine with M size (medium), 5 nodes, 2 clusters and it was COMPUTE_OPTIMIZED. We created the engine in us-east region. Firebolt does not offer us-central region. Also Firebolt does not offer L size by default that other cloud vendors offer. We got the following message when we tried to upgrade our engine.

 => ALTER ENGINE my_engine SET TYPE = "L" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2;
2025/03/15 22:46:31 ERROR query ALTER ENGINE my_engine SET TYPE = "L" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 failed: error during query execution: error during query request: Alter engine failed. 'L' node type is currently in preview. Reach out to Firebolt Support to enable 'L' node type for this account
=>

Firebolt Command Line

Also this is where we discovered that Firebolt does not offer command line. Very Strange for a database /engineer focused company.

So we build our own Firebolt Command Line !!

=> ALTER ENGINE my_engine SET TYPE = "M" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 FAMILY=CO;
ok
=> show engines;
1: engine_name: my_engine
1: engine_owner: sandhu.nurat
1: type: M
1: family: COMPUTE_OPTIMIZED
1: nodes: 5
1: clusters: 1
1: status: RUNNING
1: auto_start: 1
1: auto_stop: 20
1: initially_stopped: 0
1: url: account-1-ewafh.api.us-east-1.app.firebolt.io?engine=my_engine
1: default_database:
1: version: 4.16.8
1: last_started: 2025-03-16 20:29:35.366
1: last_stopped: 2025-03-16 20:20:19.808
1: description:
1: fbu_rate: 40
=>

You can also see tables and their various properties

=> show tables;
1: table_name: firenibble
1: table_type: BASE TABLE
1: column_count: 16
1: primary_index: f0
1: schema: CREATE TABLE "firenibble" ("f0" BIGINT NOT NULL, "f1" BIGINT NULL, "f2" BIGINT NULL, "f3" INTEGER NULL, "f4" DOUBLE PRECISION NULL, "f5" DOUBLE PRECISION NULL, "f6" DOUBLE PRECISION NULL, "f7" DOUBLE PRECISION NULL, "f8" TEXT NULL, "f9" TEXT NULL, "f10" DATE NULL, "f11" DATE NULL, "f12" DATE NULL, "f13" TEXT NULL, "f14" TEXT NULL, "f15" TEXT NULL) PRIMARY INDEX "f0"
1: number_of_rows: 1000000000
1: compressed_bytes: 216.93 GiB
1: uncompressed_bytes: 234.69 GiB
1: compression_ratio: 1
1: number_of_tablets: 1586
=>

Sweet !!

Wirekite Run

And the the fun began. We tried different combination of number of threads and number of nodes but we kept on getting the following error. Out of Memory !!

2025/03/15 22:10:46 data_loader.go:201: ERROR insert/select into table failed: error during query execution: error during query request: Out of memory - Exceeded available 22.14 GiB RAM. Consider using an engine with more RAM.
2025/03/15 22:10:46 data_loader.go:332: ERROR public.firenibble: error during query execution: error during query request: Out of memory - Exceeded available 22.14 GiB RAM. Consider using an engine with more RAM.

Finally we found a combination that worked. It was 24 threads.

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sat Mar 15 23:00:47 UTC 2025
FINISH: Sat Mar 15 23:12:24 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 11 min 37 sec

But we discovered that we were not bringing Firebolt down to its knees. Our logs told us that the loader took 5 more minutes to load the files after the mover had already moved them. As you can see here the mover finished at 23:08:54 data_mover.go:321: INFO all files processed - graceful shutdown but the loader finished at 23:12:24 data_loader.go:464: INFO all files processed - graceful shutdown.

$ grep -E 'startup|shutdown' *.log
fb_loader.log:2025/03/15 23:00:47 data_loader.go:455: INFO graceful startup
fb_loader.log:2025/03/15 23:12:24 data_loader.go:464: INFO all files processed - graceful shutdown
fb_mover.log:2025/03/15 23:00:47 data_mover.go:298: INFO graceful startup
fb_mover.log:2025/03/15 23:08:54 data_mover.go:321: INFO all files processed - graceful shutdown
pg_extract.log:2025/03/15 23:00:47 data_extractor.go:290: INFO graceful startup
pg_extract.log:2025/03/15 23:07:54 data_extractor.go:196: INFO dumped 1380 files to /mnt/benchmark/dumpdir - graceful shutdown
ziggy@instance-20250305-200657:~/benchmark/logs.20250315-231224.pg.fb.24$

Time to up the threads.

And that’s when we got our fastest run.

1 BILLION rows loaded in 8 minutes and 47 seconds.

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sat Mar 15 23:22:43 UTC 2025
FINISH: Sat Mar 15 23:31:30 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 8 min 47 sec

And the loader was lagging behind the mover by only 30 seconds. Sweet.

$ grep -E 'startup|shutdown' *.log
fb_loader.log:2025/03/15 23:22:43 data_loader.go:455: INFO graceful startup
fb_loader.log:2025/03/15 23:31:30 data_loader.go:464: INFO all files processed - graceful shutdown
fb_mover.log:2025/03/15 23:22:43 data_mover.go:298: INFO graceful startup
fb_mover.log:2025/03/15 23:30:50 data_mover.go:321: INFO all files processed - graceful shutdown
pg_extract.log:2025/03/15 23:22:43 data_extractor.go:290: INFO graceful startup
pg_extract.log:2025/03/15 23:29:52 data_extractor.go:196: INFO dumped 1380 files to /mnt/benchmark/dumpdir - graceful shutdown

And here’s the 1 Billion rows.

$ ./cmdline -c ~/wirekite/benchmark/urlfiles/fi_url.txt
INFO[0000]log.go:182 gofirebolt.(*defaultLogger).Infof Credentials cache path: /home/ziggy/.cache/firebolt/temporary_credential.json
firebolt instance connect successful
=> select count(*) as count from firenibble;
1: count: 1000000000

Or in the GUI

Airbyte Run

Running Airbyte for this connector had it’s fair share of adventures.

There were two ways to configure Airbyte to Firebolt - Inserts or S3 Bucket. We chose the bucket because it the fastest and it is what we used for our runs.

First we ran into AWS bucket permission errors. Not sure why we got that since it is configured with the same security context as our other runs. But we kept on running into this.

So we decided to move to Airbyte Cloud. And then it worked. But something to notice here - It seems airbyte first moves the data to the bucket and then loads the bucket to Firebolt. While it is loading data to the bucket it (hopefully) does not touch Firebolt so the engine possibly go to sleep, which is good since you are not charged for that time.

And here’s the Airbyte run. It took Airbyte 27 hours and 36 minutes to move 1 billion rows.

  "status" : "completed",
  "recordsSynced" : 1000000000,
  "bytesSynced" : 439710415164,
  "startTime" : 1742245104251,
  "endTime" : 1742344306406,
  "totalStats" : {
    "bytesCommitted" : 439710415164,
    "bytesEmitted" : 439710415164,
    "destinationStateMessagesEmitted" : 100000,
    "destinationWriteEndTime" : 1742344306238,
    "destinationWriteStartTime" : 1742245104389,
    "meanSecondsBeforeSourceStateMessageEmitted" : 20,
    "maxSecondsBeforeSourceStateMessageEmitted" : 100000,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 636,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 14,
    "recordsEmitted" : 1000000000,
    "recordsCommitted" : 1000000000,
    "recordsFilteredOut" : 0,
    "bytesFilteredOut" : 0,
    "replicationEndTime" : 1742344306337,
    "replicationStartTime" : 1742245104251,
    "sourceReadEndTime" : 1742343674956,
    "sourceReadStartTime" : 1742245104390,
    "sourceStateMessagesEmitted" : 100000
  },
  "streamStats" : [ {
    "streamName" : "firenibble",
    "streamNamespace" : "public",
    "stats" : {
      "bytesCommitted" : 439710415164,
      "bytesEmitted" : 439710415164,
      "recordsEmitted" : 1000000000,
      "recordsCommitted" : 1000000000,
      "recordsFilteredOut" : 0,
      "bytesFilteredOut" : 0
    }
  } ],

Also just to be clear we did use the S3 bucket method to do the load - info Using the S3 writing strategy

2025-03-17 13:58:25 info Creating status: public:firenibble - RUNNING
2025-03-17 13:58:25 info INFO main i.a.i.s.p.PostgresQueryUtils(fileNodeForIndividualStream):235 Relation filenode is for stream "public"."firenibble" is 42019
2025-03-17 13:58:25 info INFO main i.a.i.s.p.c.InitialSyncCtidIterator(createCtidQueryStatement):300 Preparing query for table: firenibble
2025-03-17 13:58:25 info INFO main i.a.i.s.p.c.InitialSyncCtidIterator(createCtidQueryStatement):309 Executing query for table firenibble: SELECT ctid::text, "f0","f1","f2","f3","f4","f5","f6","f7","f8","f9","f10","f11","f12","f13","f14","f15" FROM "public"."firenibble" WHERE ctid > ?::tid AND ctid <= ?::tid with bindings (0,0) and (131072,0)
2025-03-17 13:58:25 info INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(initialize):24 Set initial fetch size: 10 rows
2025-03-17 13:58:26 info INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(accept):33 Set new fetch size: 173376 rows
2025-03-17 13:58:26 info Using the S3 writing strategy
2025-03-17 13:58:26 info INFO main i.a.c.d.j.s.TwoStageSizeEstimator$Companion(getTargetBufferByteSize):80 Max memory limit: 1610612736, JDBC buffer size: 966367642
2025-03-17 13:58:26 info Stream firenibble is wiped.
2025-03-17 13:58:26 info INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(accept):33 Set new fetch size: 637866 rows
2025-03-17 13:58:27 info INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(accept):33 Set new fetch size: 636606 rows
2025-03-17 13:58:27 info INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(accept):33 Set new fetch size: 635350 rows
2025-03-17 13:58:28 info INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(accept):33 Set new fetch size: 631613 rows

I started wondering why did it take so long while it moved the same table to Snowflake in 12 hours. And I think the answer is this. Airbyte apparently creates the migrated table as 3 columns.

_airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed. The column type in Firebolt is VARCHAR.
_airbyte_emitted_at: a timestamp representing when the event was pulled from the data source. The column type in Firebolt is TIMESTAMP.
_airbyte_data: a json blob representing the event data. The column type in Firebolt is VARCHAR but can be parsed with JSON functions.

So all the row data is stored as a JSON. My guess is Airbyte is spending hell lot of time making a JSON out of each row.

Which is what we saw in our Firebolt database.

By the way did you notice the query to do count ran in 0.20 seconds. Sweet !!

Oracle to Firebolt - 10 million ops (inserts + updates + deletes) - 5 mins 50 secs

Wirekite — Sat, 07 Jun 2025 07:00:00 GMT

So we set up a Change Data Capture replication pipeline from Oracle to Firebolt and wanted to benchmark how it performs under a “typical” OLTP load. This article discusses the various characteristics of the benchmark.

Benchmark Details

It has multiple tables.
It has some inserts.
It has somewhat more updates.
It has some deletes.
It has small transactions with lot of commits, mostly single row operations and commits.

Given the above characteristics we decided on doing the following for the benchmark.

We will have 10 tables.
We will have 2 million single row inserts.
We will have 6 million single row updates.
We will have 2 million single row deletes.
We will have auto commit on, causing all the above operations to have corresponding commit.

Table Structure

We will have 10 tables called firenibble 1 to 10.

CREATE TABLE app_db.firenibble1 (
  f0 number(38) NOT NULL,
  f1 number(38) ,
  f2 number(38) ,
  f3 int ,
  f4 float ,
  f5 float ,
  f6 float ,
  f7 float ,
  f8 varchar2(128) ,
  f9 varchar2(128) ,
  f10 date ,
  f11 date ,
  f12 date ,
  f13 varchar2(128) ,
  f14 varchar2(128) ,
  f15 varchar2(128) ,
  primary key (f0)
);

Insert

Insert will essentially be new primary key insert.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Update

Update will be updating of two columns on a primary key value.

update firenibble1 set f2 = 280396830734761092, f9 = 'n6ChQjN9dfxIl6nc6SNgIzSahfeXfAef' where f0 = 19998;

Delete

Delete will be essentially a primary key delete.

delete from firenibble1 where f0 = 19998;

DML Operations Sequence

So the exact sequence will look like this. I know we could have been more granular mixing the inserts, updates and deletes even more, and as we do more benchmarks we might try that strategy but for starters this looked pretty good. Also we wanted to see how our replication performed with large number of insert only, update only and delete only loads. Also note that there is no relationships between the load and the archivelogs, in the sense there is no alter system switch logfile happening anywhere causing the inserts, updates and deletes to cross binary log boundaries.

1 million inserts
3 million updates
1 million deletes
1 million inserts
3 million updates
1 million deletes.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble2 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble3 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble4 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble5 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble6 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble7 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble8 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble9 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble10 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Benchmark Sequence

A choice we had was to make was do one of the following

Have the wirekite replication running. Run the load and wait for the last event of the load to appear on the target. Measure the time between when the load started and the last event appeared.
Run the load and note down the binlog number and position at which the load started and wait for the load to finish. Now start the wirekite replication from the starting binlog number and position and wait for the last event of the load to appear on the target. Measure the time between when the wirekite replication started and the last event appeared on the target.

Machine Configuration

For Firebolt we created a Medium Size Warehouse.

=> show engines;
1: engine_name: my_engine
1: engine_owner: gjkemnitz
1: type: M
1: family: COMPUTE_OPTIMIZED
1: nodes: 5
1: clusters: 1
1: status: RUNNING
1: auto_start: 1
1: auto_stop: 20
1: initially_stopped: 0
1: url: account-1-fxrjk.api.us-east-1.app.firebolt.io?engine=my_engine
1: default_database:
1: version: 4.21.9
1: last_started: 2025-06-14 04:28:27.824
1: last_stopped: 2025-06-14 03:58:10.993
1: description:
1: fbu_rate: 40

I wish we could automatically create a Large Size Warehouse but we got the following error.

=> ALTER ENGINE my_engine SET TYPE = "L";
2025/06/14 04:49:00 ERROR query ALTER ENGINE my_engine SET TYPE = "L" failed: error during query execution: error during query request: Alter engine failed. 'L_COMPUTE_OPTIMIZED' node type is currently in preview. Reach out to Firebolt Support to enable 'L_COMPUTE_OPTIMIZED' node type for this account

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

Oracle Run Around

We wanted to mention this a bit to talk about the uphill challenge that was getting oracle to be up and running and importing data into it.

We realized that some given environment variable has to be set to make sure the GUI runs as intended. Once we did that we got the right GUI and got Oracle installed.

And then came the challenge of configuring the database with sensible values so that you can actually load a reasonable amount of data in it. We had to configure

Configure and start the LISTENER. Understand LISTENER.ORA and TNSNAMES.ORA.
Figure out the differences between PFILE and SPFILE and the workings of SCOPE.
Turn off Automatic Shared Memory Management - MEMORY_TARGET = 0
Turn off Automatic Management - SGA_TARGET = 0
Set SGA_MAX_SIZE
Set DB_CACHE_SIZE
Set SHARED_POOL_SIZE
Set LARGE_POOL_SIZE
Modify a few kernel parameters (SHMMAX, SHMALL, file open limits), modify sysctl.conf and limits.conf and reboot the instance a few times.
Create a separate TABLESPACE to hold the table with the right number and size of data files and make sure they extend automatically.
Grant privileges on this TABLESPACE to the right USER.
Create the table with the right TABLESPACE for data but also the right TABLESPACE for INDEX otherwise it will create index in the default tablespace.
Figure out SQL LOADER to load the data from a csv into the table. Make sure you have the right commit frequency.
Monitor DBA_FREE_SPACE and add data files when it comes close.
Play with V$SESSION and V$SESSION_WAIT to see what is happening.
Play with EXPLAIN PLANS since Oracle gives a thousand ways to see the actual explain plan.

No Airbyte Run

We could not do an airbyte run since it requires an enterprise level connector. So this is just our benchmark.

Wirekite Run

So how long did it take for wirekite to move 10 million changes — 5 minutes and 50 seconds.

cat wirekite_cdc.log | grep -E 'START|FINISH|ELAPSED'
START: Sat Jun 14 04:40:51 UTC 2025
FINISH: Sat Jun 14 04:46:41 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 5 min 50 sec

MySQL to Snowflake - 1B rows - 8 minutes 56 seconds

Wirekite — Fri, 06 Jun 2025 07:00:00 GMT

So we did another benchmark this time with MySQL to Snowflake to demonstrate how Wirekite sizes up in the enterprise data migration space. Again we decided to compare the benchmark with Airbyte since it is one of the most recognized data migration enterprise software out there.

Benchmark Details

Table

First of all lets talk about the table structure. We have used the same table that we used in other benchmarks. The table has 16 columns, one of them is a numeric primary key, few columns are numbers of various types, few strings and few dates. This makes the table pretty simple to be absorbed by most databases.

The table also has 1 Billion rows.

We want to make it easy and standardized across various other benchmarks and databases.

CREATE TABLE `firenibble` (
  `f0` bigint NOT NULL,
  `f1` bigint DEFAULT NULL,
  `f2` bigint DEFAULT NULL,
  `f3` int DEFAULT NULL,
  `f4` float DEFAULT NULL,
  `f5` float DEFAULT NULL,
  `f6` float DEFAULT NULL,
  `f7` float DEFAULT NULL,
  `f8` varchar(256) DEFAULT NULL,
  `f9` varchar(256) DEFAULT NULL,
  `f10` date DEFAULT NULL,
  `f11` date DEFAULT NULL,
  `f12` date DEFAULT NULL,
  `f13` varchar(256) DEFAULT NULL,
  `f14` varchar(256) DEFAULT NULL,
  `f15` varchar(256) DEFAULT NULL,
  PRIMARY KEY (`f0`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci DATA DIRECTORY='/mnt/benchmark/mysqldata/'

And here’s the 1 billion rows.

mysql> select count(*) from firenibble;
+------------+
| count(*)   |
+------------+
| 1000000000 |
+------------+
1 row in set (13 min 4.70 sec)

mysql>

Machine Configuration

For MySQL we created an n2-standard-64 GCP instance with all the standard configurations - 64 CPU’s and 256 GB RAM in the us-central region. We installed Ubuntu version 24 and MySQL version 8.

For Snowflake we created a Large size warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

Airbyte Run

We created the firenibble table with Full Refresh Overwrite and made sure the table did not exist in the target, thus forcing a full sync of the whole table.

And then we ran the benchmark only to get the following message after waiting for a long time.

After waiting and watching for almost 18 hours and transferring 103 million records, we did not have the courage to run the benchmark again. Turns out the snowflake free trial that we used to do the benchmark had expired. So we decided to accept this as a technically invalid benchmark, but since the rate of transfer was not changing drastically, we decided to publish it.

The logs

2025-03-10 23:46:48 info sync summary: {
  "status" : "failed",
  "recordsSynced" : 103123546,
  "bytesSynced" : 41680061203,
  "startTime" : 1741644491842,
  "endTime" : 1741650408147,
  "totalStats" : {
    "bytesCommitted" : 41680061203,
    "bytesEmitted" : 43332279893,
    "destinationStateMessagesEmitted" : 21,
    "destinationWriteEndTime" : 0,
    "destinationWriteStartTime" : 1741644491938,
    "meanSecondsBeforeSourceStateMessageEmitted" : 376,
    "maxSecondsBeforeSourceStateMessageEmitted" : 21,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 73,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 59,
    "recordsEmitted" : 107200542,
    "recordsCommitted" : 103123546,
    "recordsFilteredOut" : 0,
    "bytesFilteredOut" : 0,
    "replicationEndTime" : 1741650408144,
    "replicationStartTime" : 1741644491842,
    "sourceReadEndTime" : 0,
    "sourceReadStartTime" : 1741644491939,
    "sourceStateMessagesEmitted" : 21
  }

But our favorite message in the logs was the following !

"internalMessage" : "net.snowflake.client.jdbc.SnowflakeSQLException: Your account is suspended due to lack of payment method.",

Stack Trace: net.snowflake.client.jdbc.SnowflakeSQLException: Your free trial has ended and all of your virtual warehouses have been suspended. Add billing information in the Snowflake web UI to continue using the full set of Snowflake features.

You are poor !! Yes we are.

Wirekite Run

After rebooting the MySQL GCP instance to flush the buffer pool and any other remnants of the previous run we ran the same benchmark.

Everything else being same we got the following results.

1 billion rows moved under 9 minutes !!

# cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Mon Mar 10 03:06:20 UTC 2025
FINISH: Mon Mar 10 03:15:16 UTC 2025
ELAPSED: 0 hrs 8 min 56 sec

The billion rows in Snowflake.

#COMPUTE_WH@TEST.APP_DB>select count(*) from FIRENIBBLE;
+------------+
|   COUNT(*) |
|------------|
| 1000000000 |
+------------+
1 Row(s) produced. Time Elapsed: 0.242s
#COMPUTE_WH@TEST.APP_DB>
Goodbye!

MySQL to Snowflake - 10 million ops (inserts + updates + deletes) - 2 mins 41 secs

Wirekite — Fri, 06 Jun 2025 07:00:00 GMT

So we set up a Change Data Capture replication pipeline from MySQL to Snowflake and wanted to benchmark how it performs under a “typical” OLTP load. This article discusses the various characteristics of the benchmark and how did we fare compared to standard tools out there.

Benchmark Details

It has multiple tables.
It has some inserts.
It has somewhat more updates.
It has some deletes.
It has small transactions with lot of commits, mostly single row operations and commits.

Given the above characteristics we decided on doing the following for the benchmark.

We will have 10 tables.
We will have 2 million single row inserts.
We will have 6 million single row updates.
We will have 2 million single row deletes.
We will have auto commit on, causing all the above operations to have corresponding commit.

Table Structure

We will have 10 tables called firenibble 1 to 10.

CREATE TABLE public.firenibble1 (
  f0 bigint NOT NULL,
  f1 bigint DEFAULT NULL,
  f2 bigint DEFAULT NULL,
  f3 int DEFAULT NULL,
  f4 double DEFAULT NULL,
  f5 double DEFAULT NULL,
  f6 double DEFAULT NULL,
  f7 double DEFAULT NULL,
  f8 varchar(128) DEFAULT NULL,
  f9 varchar(128) DEFAULT NULL,
  f10 date DEFAULT NULL,
  f11 date DEFAULT NULL,
  f12 date DEFAULT NULL,
  f13 varchar(128) DEFAULT NULL,
  f14 varchar(128) DEFAULT NULL,
  f15 varchar(128) DEFAULT NULL,
  PRIMARY KEY (f0)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Insert

Insert will essentially be new primary key insert.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Update

Update will be updating of two columns on a primary key value.

update firenibble1 set f2 = 280396830734761092, f9 = 'n6ChQjN9dfxIl6nc6SNgIzSahfeXfAef' where f0 = 19998;

Delete

Delete will be essentially a primary key delete.

delete from firenibble1 where f0 = 19998;

DML Operations Sequence

Then came the question of how do we order the load so that we can rerun the load over and over again and never have to clean up the previous load. So we decided that the load will be such that we will start with 0 rows in all 10 tables, do a bunch of inserts, do a bunch of updates, do a bunch of deletes and end up with 0 rows in all tables. This kept the load and the tables rather clean. So the exact sequence will look like this. I know we could have been more granular mixing the inserts, updates and deletes even more, and as we do more benchmarks we might try that strategy but for starters this looked pretty good. Also we wanted to see how our replication performed with large number of insert only, update only and delete only loads. Also note that there is no relationships between the load and the binlogs, in the sense there is no flush binary logs happening anywhere causing the inserts, updates and deletes to cross binary log boundaries.

1 million inserts
3 million updates
1 million deletes
1 million inserts
3 million updates
1 million deletes.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble2 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble3 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble4 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble5 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble6 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble7 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble8 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble9 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble10 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Benchmark Sequence

A choice we had was to make was do one of the following

Have the wirekite replication running. Run the load and wait for the last event of the load to appear on the target. Measure the time between when the load started and the last event appeared.
Run the load and note down the binlog number and position at which the load started and wait for the load to finish. Now start the wirekite replication from the starting binlog number and position and wait for the last event of the load to appear on the target. Measure the time between when the wirekite replication started and the last event appeared on the target.

Machine Configuration

For MySQL we created an n2-standard-8 GCP instance with all the standard configurations - 8 CPU’s and 32 GB RAM in the us-central region. We installed Ubuntu version 24 and MySQL version 8. Our binlog size was 100 MB.

For Snowflake we created a Large Size Warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

Airbyte Run

We created the MySQL to Snowflake connection in Airbyte and made sure we picked the right options. For MySQL we picked the CDC option for syncing since that’s what airbyte recommended.

We also picked the option which creates the mirror copy of source data on the target since that’s what we are doing.

We also picked the Incremental | Append + Deduped option, since it is what gives us the mirror copy of the data on the target.

For Snowflake we checked the Merge option since we are also presenting data on the target completely as a mirror of the source, i.e de-duplicated.

And then we let the sync run.

And the sync finished successfully !

The logs

2025-04-18 05:03:14 info sync summary: {
  "status" : "completed",
  "recordsSynced" : 10000000,
  "bytesSynced" : 5646273037,
  "startTime" : 1744951533820,
  "endTime" : 1744952594325,
  "totalStats" : {
    "bytesCommitted" : 5646273037,
    "bytesEmitted" : 5646273037,
    "destinationStateMessagesEmitted" : 10000,
    "destinationWriteEndTime" : 1744952594192,
    "destinationWriteStartTime" : 1744951533994,
    "meanSecondsBeforeSourceStateMessageEmitted" : 48,
    "maxSecondsBeforeSourceStateMessageEmitted" : 1000,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 302,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 219,
    "recordsEmitted" : 10000000,
    "recordsCommitted" : 10000000,
    "recordsFilteredOut" : 0,
    "bytesFilteredOut" : 0,
    "replicationEndTime" : 1744952594209,
    "replicationStartTime" : 1744951533820,
    "sourceReadEndTime" : 1744952549452,
    "sourceReadStartTime" : 1744951533995,
    "sourceStateMessagesEmitted" : 10000
  },

The final tally —

Airbyte took 17 minutes and 54 seconds to finish the run !!

Note that you do see an earlier sync also. That sync was the initial sync that was used to create initial tables (with 0 records) on the target so that we can then do the replication run.

Wirekite Run

So then we did the same exact run in wirekite.

Wirekite moved the same amount of changes in 2 minutes and 41 seconds.

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sat Apr 19 05:46:07 UTC 2025
FINISH: Sat Apr 19 05:48:48 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 2 min 41 sec

MySQL to Firebolt - 1B rows - 9 minutes 53 seconds

Wirekite — Thu, 05 Jun 2025 07:00:00 GMT

So we did our second benchmark for Firebolt, this time from MySQL to demonstrate the speed of Wirekite in the enterprise data migration space from one database to another. We decided to not to compare this with Airbyte since we ran into some errors. Also we think the Airbyte Postgres to Firebolt benchmark will give you a good idea where it generally stands.

To see the Postgres benchmark please visit - https://benchmarks.wirekite.io/postgres-to-firebolt-1b-rows-8-minutes-47-seconds.

Benchmark Details

Table

As always we used the same firenibble table with a billion rows It has 15 columns and has all the datatypes - numbers of various shapes and sizes, some strings and some dates. This makes sure that these datatypes fit in most databases. Also we have used the same table everywhere to make sure our benchmarks stack up nicely in comparison with each other.

 CREATE TABLE `firenibble` (
  `f0` bigint NOT NULL,
  `f1` bigint DEFAULT NULL,
  `f2` bigint DEFAULT NULL,
  `f3` int DEFAULT NULL,
  `f4` float DEFAULT NULL,
  `f5` float DEFAULT NULL,
  `f6` float DEFAULT NULL,
  `f7` float DEFAULT NULL,
  `f8` varchar(256) DEFAULT NULL,
  `f9` varchar(256) DEFAULT NULL,
  `f10` date DEFAULT NULL,
  `f11` date DEFAULT NULL,
  `f12` date DEFAULT NULL,
  `f13` varchar(256) DEFAULT NULL,
  `f14` varchar(256) DEFAULT NULL,
  `f15` varchar(256) DEFAULT NULL,
  PRIMARY KEY (`f0`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci DATA DIRECTORY='/mnt/benchmark/mysqldata/'

And here’s the 1 billion rows.

mysql> select count(*) from firenibble;
+------------+
| count(*)   |
+------------+
| 1000000000 |
+------------+
1 row in set (13 min 4.70 sec)

mysql>

Machine Configurations

For MySQL we created an n2-standard-64 GCP instance with all the standard configurations - 64 CPU’s and 256 GB RAM in the us-central region. We installed Ubuntu version 24 and MySQL version 8.

 => ALTER ENGINE my_engine SET TYPE = "L" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2;
2025/03/15 22:46:31 ERROR query ALTER ENGINE my_engine SET TYPE = "L" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 failed: error during query execution: error during query request: Alter engine failed. 'L' node type is currently in preview. Reach out to Firebolt Support to enable 'L' node type for this account
=>

Firebolt Command Line

Also this is where we discovered that Firebolt does not offer command line. Very Strange for a database /engineer focused company.

So we build our own Firebolt Command Line !!

=> ALTER ENGINE my_engine SET TYPE = "M" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 FAMILY=CO;
ok
=> show engines;
1: engine_name: my_engine
1: engine_owner: sandhu.nurat
1: type: M
1: family: COMPUTE_OPTIMIZED
1: nodes: 5
1: clusters: 1
1: status: RUNNING
1: auto_start: 1
1: auto_stop: 20
1: initially_stopped: 0
1: url: account-1-ewafh.api.us-east-1.app.firebolt.io?engine=my_engine
1: default_database:
1: version: 4.16.8
1: last_started: 2025-03-16 20:29:35.366
1: last_stopped: 2025-03-16 20:20:19.808
1: description:
1: fbu_rate: 40
=>

You can also see tables and their various properties

=> show tables;
1: table_name: firenibble
1: table_type: BASE TABLE
1: column_count: 16
1: primary_index: f0
1: schema: CREATE TABLE "firenibble" ("f0" BIGINT NOT NULL, "f1" BIGINT NULL, "f2" BIGINT NULL, "f3" INTEGER NULL, "f4" DOUBLE PRECISION NULL, "f5" DOUBLE PRECISION NULL, "f6" DOUBLE PRECISION NULL, "f7" DOUBLE PRECISION NULL, "f8" TEXT NULL, "f9" TEXT NULL, "f10" DATE NULL, "f11" DATE NULL, "f12" DATE NULL, "f13" TEXT NULL, "f14" TEXT NULL, "f15" TEXT NULL) PRIMARY INDEX "f0"
1: number_of_rows: 1000000000
1: compressed_bytes: 216.93 GiB
1: uncompressed_bytes: 234.69 GiB
1: compression_ratio: 1
1: number_of_tablets: 1586
=>

Airbyte Run

We tried Airbyte but it failed on us with errors that we did not how to troubleshoot.

But there is one very interesting thing we found. Airbyte is essentially using Debezium for its MySQL extractor. We found that to be pretty strange since it competes with Debezium.

ziggy@instance-20240626-215025:~/Downloads$ cat mysql___firebolt_logs_3_txt.txt | grep -i debezium-engine
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    connector.class = io.debezium.connector.mysql.MySqlConnector
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    snapshot.locking.mode = none
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    max.queue.size = 8192
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    database.sslMode = preferred
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    include.schema.changes = false
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    schema.history.internal.store.only.captured.databases.ddl = true
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    schema.history.internal.file.filename = /tmp/airbyte-debezium-state11123047976539933388/dbhistory.dat
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    value.converter.replace.null.with.default = false
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    topic.prefix = public
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    decimal.handling.mode = string
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    binary.handling.mode = base64
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    offset.storage.file.filename = /tmp/airbyte-debezium-state11123047976539933388/offset.dat
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    errors.retry.delay.initial.ms = 299
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    converters = boolean,temporal
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    value.converter = org.apache.kafka.connect.json.JsonConverter
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    key.converter = org.apache.kafka.connect.json.JsonConverter
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    database.sessionVariables = autocommit=0
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    database.user = ziggy
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    database.dbname = public
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    offset.storage = org.apache.kafka.connect.storage.FileOffsetBackingStore
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    database.server.id = 5511
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    max.queue.size.in.bytes = 268435456
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    database.useCursorFetch = true
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    boolean.type = io.airbyte.integrations.source.mysql.MySqlSourceCdcBooleanConverter
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    time.precision.mode = adaptive_time_microseconds
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    temporal.type = io.airbyte.integrations.source.mysql.MySqlSourceCdcTemporalConverter
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    errors.retry.delay.max.ms = 300
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    offset.flush.timeout.ms = 5000
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    heartbeat.interval.ms = 10000
2025-03-16 23:53:05 info INFO debezium-engine i.d.c.c.BaseSourceTask(lambda$start$0):247    schema.history.internal.skip.unparseable.ddl = true

Wirekite Run

After running into a bunch of Out of Memory error, we found the right combination of threads.

Out of memory - Exceeded available 22.14 GiB RAM. Consider using an engine with more RAM.

And we got a successful run.

1 BILLION rows moved in 9 minutes and 53 seconds.

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sun Mar 16 00:57:22 UTC 2025
FINISH: Sun Mar 16 01:07:15 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 9 min 53 sec

An interesting to note here is how Firebolt loader lagged the mover. We want the lag to be minimum which essentially makes sure that we are using Firebolt to its full capacity. As you can see the loader - 01:07:15 data_loader.go:464: INFO all files processed - graceful shutdown - lagged the mover - 01:06:21 data_mover.go:321: INFO all files processed - graceful shutdown - by about a minute. Not bad !!

$ grep -E 'startup|shutdown' *.log
fb_loader.log:2025/03/16 00:57:23 data_loader.go:455: INFO graceful startup
fb_loader.log:2025/03/16 01:07:15 data_loader.go:464: INFO all files processed - graceful shutdown
fb_mover.log:2025/03/16 00:57:22 data_mover.go:298: INFO graceful startup
fb_mover.log:2025/03/16 01:06:21 data_mover.go:321: INFO all files processed - graceful shutdown
my_extract.log:2025/03/16 00:57:22 data_extractor_parallel.go:328: INFO graceful startup
my_extract.log:2025/03/16 01:06:10 data_extractor_parallel.go:221: INFO dumped 1250 files to /mnt/benchmark/dumpdir - graceful shutdown

And here’s the 1 billion rows.

INFO[0000]log.go:182 gofirebolt.(*defaultLogger).Infof Credentials cache path: /home/ziggy/.cache/firebolt/temporary_credential.json
firebolt instance connect successful
=> select count(*) as count from firenibble;
1: count: 1000000000

And on the GUI

MySQL to Firebolt - 10 million ops (inserts + updates + deletes) - 0 mins 53 secs

Wirekite — Thu, 05 Jun 2025 07:00:00 GMT

So we set up a Change Data Capture replication pipeline from MySQL to Firebolt and wanted to benchmark how it performs under a “typical” OLTP load. This article discusses the various characteristics of the benchmark and how did we fare compared to standard tools out there.

Benchmark Details

It has multiple tables.
It has some inserts.
It has somewhat more updates.
It has some deletes.
It has small transactions with lot of commits, mostly single row operations and commits.

Given the above characteristics we decided on doing the following for the benchmark.

We will have 10 tables.
We will have 2 million single row inserts.
We will have 6 million single row updates.
We will have 2 million single row deletes.
We will have auto commit on, causing all the above operations to have corresponding commit.

Table Structure

We will have 10 tables called firenibble 1 to 10.

CREATE TABLE public.firenibble1 (
  f0 bigint NOT NULL,
  f1 bigint DEFAULT NULL,
  f2 bigint DEFAULT NULL,
  f3 int DEFAULT NULL,
  f4 double DEFAULT NULL,
  f5 double DEFAULT NULL,
  f6 double DEFAULT NULL,
  f7 double DEFAULT NULL,
  f8 varchar(128) DEFAULT NULL,
  f9 varchar(128) DEFAULT NULL,
  f10 date DEFAULT NULL,
  f11 date DEFAULT NULL,
  f12 date DEFAULT NULL,
  f13 varchar(128) DEFAULT NULL,
  f14 varchar(128) DEFAULT NULL,
  f15 varchar(128) DEFAULT NULL,
  PRIMARY KEY (f0)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Insert

Insert will essentially be new primary key insert.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Update

Update will be updating of two columns on a primary key value.

update firenibble1 set f2 = 280396830734761092, f9 = 'n6ChQjN9dfxIl6nc6SNgIzSahfeXfAef' where f0 = 19998;

Delete

Delete will be essentially a primary key delete.

delete from firenibble1 where f0 = 19998;

DML Operations Sequence

So the exact sequence will look like this. I know we could have been more granular mixing the inserts, updates and deletes even more, and as we do more benchmarks we might try that strategy but for starters this looked pretty good. Also we wanted to see how our replication performed with large number of insert only, update only and delete only loads. Also note that there is no relationships between the load and the binlogs, in the sense there is no flush binary logs happening anywhere causing the inserts, updates and deletes to cross binary log boundaries.

1 million inserts
3 million updates
1 million deletes
1 million inserts
3 million updates
1 million deletes.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble2 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble3 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble4 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble5 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble6 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble7 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble8 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble9 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble10 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Benchmark Sequence

A choice we had was to make was do one of the following

Have the wirekite replication running. Run the load and wait for the last event of the load to appear on the target. Measure the time between when the load started and the last event appeared.
Run the load and note down the binlog number and position at which the load started and wait for the load to finish. Now start the wirekite replication from the starting binlog number and position and wait for the last event of the load to appear on the target. Measure the time between when the wirekite replication started and the last event appeared on the target.

Machine Configuration

For MySQL we created an n2-standard-16 GCP instance with all the standard configurations - 16 CPU’s and 64 GB RAM in the us-central region. We installed Ubuntu version 24 and MySQL version 8. Our binlog size was 100 MB.

For Snowflake we created a Medium Size Warehouse.

=> show engines;
1: engine_name: my_engine
1: engine_owner: gjkemnitz
1: type: M
1: family: COMPUTE_OPTIMIZED
1: nodes: 5
1: clusters: 1
1: status: RUNNING
1: auto_start: 1
1: auto_stop: 20
1: initially_stopped: 0
1: url: account-1-fxrjk.api.us-east-1.app.firebolt.io?engine=my_engine
1: default_database:
1: version: 4.21.9
1: last_started: 2025-06-14 04:28:27.824
1: last_stopped: 2025-06-14 03:58:10.993
1: description:
1: fbu_rate: 40

We would have loved to upgrade the size to Large but we got the following error.

=> ALTER ENGINE my_engine SET TYPE = "L";
2025/06/14 04:49:00 ERROR query ALTER ENGINE my_engine SET TYPE = "L" failed: error during query execution: error during query request: Alter engine failed. 'L_COMPUTE_OPTIMIZED' node type is currently in preview. Reach out to Firebolt Support to enable 'L_COMPUTE_OPTIMIZED' node type for this account.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

Wirekite Run

So how long did it take to move 10 million operations from MySQL to Firebolt — 0 minutes and 53 seconds !

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sat Jun 14 05:33:14 UTC 2025
FINISH: Sat Jun 14 05:34:07 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 0 min 53 sec

Oracle to Snowflake - 1B rows - 6 minutes 9 seconds

Wirekite — Wed, 04 Jun 2025 07:00:00 GMT

So one more benchmark except this time with a really expensive enterprise grade database.

We could not compare this benchmark with Airbyte since Airbyte does not allow an open source connector for Oracle. Entirely understandable. When we tried to configure the connector after spending a long time installing Airbyte we got the following message. Would have been nice to know this earlier.

Benchmark Details

Since we could not do an Airbyte benchmark we decided to play a little bit with oracle and see how far we can scale our benchmarks as we increased the source Oracle machine configuration, the target Snowflake configurations and the number of wirekite threads.

Table

As usual we used our standard firenibble table.

The table also has 1 Billion rows.

CREATE TABLE app_db.firenibble (
    f0    number(38)    NOT NULL,
    f1    number(38),
    f2    number(38),
    f3    int,
    f4    float,
    f5    float,
    f6    float,
    f7    float,
    f8    varchar2(128),
    f9    varchar2(128),
    f10    date,
    f11    date,
    f12    date,
    f13    varchar2(128),
    f14    varchar2(128),
    f15    varchar2(128),
    PRIMARY KEY (f0)
);

The 1 billion rows.

SQL> select count(*) from app_db.firenibble;

  COUNT(*)
----------
1000000000

1 row selected.

SQL>

Oracle Stuff

We wanted to mention this a bit to talk about the uphill challenge that was getting oracle to be up and running and importing data into it.

We realized that some given environment variable has to be set to make sure the GUI runs as intended. Once we did that we got the right GUI and got Oracle installed.

And then came the challenge of configuring the database with sensible values so that you can actually load a reasonable amount of data in it. We had to configure

Configure and start the LISTENER. Understand LISTENER.ORA and TNSNAMES.ORA.
Figure out the differences between PFILE and SPFILE and the workings of SCOPE.
Turn off Automatic Shared Memory Management - MEMORY_TARGET = 0
Turn off Automatic Management - SGA_TARGET = 0
Set SGA_MAX_SIZE
Set DB_CACHE_SIZE
Set SHARED_POOL_SIZE
Set LARGE_POOL_SIZE
Modify a few kernel parameters (SHMMAX, SHMALL, file open limits), modify sysctl.conf and limits.conf and reboot the instance a few times.
Create a separate TABLESPACE to hold the table with the right number and size of data files and make sure they extend automatically.
Grant privileges on this TABLESPACE to the right USER.
Create the table with the right TABLESPACE for data but also the right TABLESPACE for INDEX otherwise it will create index in the default tablespace.
Figure out SQL LOADER to load the data from a csv into the table. Make sure you have the right commit frequency.
Monitor DBA_FREE_SPACE and add data files when it comes close.
Play with V$SESSION and V$SESSION_WAIT to see what is happening.
Play with EXPLAIN PLANS since Oracle gives a thousand ways to see the actual explain plan.

And phew !! You have an oracle database and the table with the data that you desire.

So how does Oracle look like when it is completely pushed to its seams. This is how. As you can see either the wait event is db file sequential read - which means it is going through the primary key index - or it is SQL*Net more data to client - which means it is sending data to the client.

INST_ID                              START                                               WAIT
:SID    USERNAME    MACHINE                  D HH:MM:SS  SQL_ID       MODULE        EVENT                SEQ#  (SEC)
------- --------------- ---------------------------------------- ----------- ------------- ---------------- --------------------------------- ------ ------
1:676    WIREKITE    instance-20240906-204153         +0 00:00:00 975qxtgwmxfp6 runquery@instan  SQL*Net more data to client      122      0
1:604    WIREKITE    instance-20240906-204153         +0 00:00:01 3wvvjvptdzx33 runquery@instan  db file sequential read         783      0
1:532    WIREKITE    instance-20240906-204153         +0 00:00:01 5sxg963sxm3ay runquery@instan  ON CPU                1353      0
1:556    WIREKITE    instance-20240906-204153         +0 00:00:01 8ywzssbgay2rc runquery@instan  db file sequential read        1045      0
1:628    WIREKITE    instance-20240906-204153         +0 00:00:01 dtks5wx6dc7ug runquery@instan  SQL*Net more data to client      616      0
1:580    WIREKITE    instance-20240906-204153         +0 00:00:01 fvf8wxqwx30w3 runquery@instan  ON CPU                 906      0
1:508    WIREKITE    instance-20240906-204153         +0 00:00:01 gjvnup94j6nq7 runquery@instan  ON CPU                1297      0
1:412    WIREKITE    instance-20240906-204153         +0 00:00:02 1y5crzp0bvksm runquery@instan  ON CPU                1681      0
1:316    WIREKITE    instance-20240906-204153         +0 00:00:02 4kthrfxrk0d62 runquery@instan  SQL*Net more data to client     1785      0
1:364    WIREKITE    instance-20240906-204153         +0 00:00:02 5tm23gtarzn0d runquery@instan  ON CPU                1833      0
1:459    WIREKITE    instance-20240906-204153         +0 00:00:02 6gzbbtnmsxcam runquery@instan  SQL*Net more data to client     1527      0
1:341    WIREKITE    instance-20240906-204153         +0 00:00:02 78u3msqm33wv8 runquery@instan  ON CPU                1736      0
1:245    WIREKITE    instance-20240906-204153         +0 00:00:02 7xv7s23bsjxwa runquery@instan  ON CPU                2029      0
1:221    WIREKITE    instance-20240906-204153         +0 00:00:02 8ask5tjwyqg95 runquery@instan  db file sequential read        2499      0
1:291    WIREKITE    instance-20240906-204153         +0 00:00:02 8qna8nwchb7yx runquery@instan  ON CPU                2528      0
1:197    WIREKITE    instance-20240906-204153         +0 00:00:02 94b8wkyb6vyxn runquery@instan  ON CPU                2597      0
1:172    WIREKITE    instance-20240906-204153         +0 00:00:02 b1y9um3f5gz99 runquery@instan  SQL*Net more data to client     2672      0
1:268    WIREKITE    instance-20240906-204153         +0 00:00:02 cg0s1hcp4k72z runquery@instan  ON CPU                2452      0
1:436    WIREKITE    instance-20240906-204153         +0 00:00:02 d5r7aba9hv1dt runquery@instan  SQL*Net more data to client     1549      0
1:388    WIREKITE    instance-20240906-204153         +0 00:00:02 dpn54kc4j6bdt runquery@instan  ON CPU                1586      0
1:484    WIREKITE    instance-20240906-204153         +0 00:00:02 fgyhxt1tyzj2z runquery@instan  ON CPU                1475      0
1:76    WIREKITE    instance-20240906-204153         +0 00:00:02 ghvu59jj9r9r4 runquery@instan  ON CPU                1957      0
1:52    WIREKITE    instance-20240906-204153         +0 00:00:03 129kf42rfscj3 runquery@instan  ON CPU                3159      0
1:100    WIREKITE    instance-20240906-204153         +0 00:00:03 2kx656wz4gsd8 runquery@instan  ON CPU                3426      0
1:124    WIREKITE    instance-20240906-204153         +0 00:00:03 7782fuuud928k runquery@instan  ON CPU                2988      0
1:3    WIREKITE    instance-20240906-204153         +0 00:00:03 brjxucsns1r3c runquery@instan  ON CPU                3784      0
1:1467    WIREKITE    instance-20240906-204153         +0 00:00:03 cg5746pt2fnrf runquery@instan  SQL*Net more data to client     3960      0
1:1444    WIREKITE    instance-20240906-204153         +0 00:00:03 ck8hth7q8dfm0 runquery@instan  db file sequential read        3878      0
1:27    WIREKITE    instance-20240906-204153         +0 00:00:03 fqjtzvs9qbyf9 runquery@instan  ON CPU                3170      0
1:1491    WIREKITE    instance-20240906-204153         +0 00:00:03 g0p4s6wp1sfvg runquery@instan  ON CPU                3254      0
1:1323    WIREKITE    instance-20240906-204153         +0 00:00:04 4188ktumx0b49 runquery@instan  SQL*Net more data to client     5191      0
1:1394    WIREKITE    instance-20240906-204153         +0 00:00:04 7f4mcwydwm3ky runquery@instan  db file sequential read        4169      0
1:1370    WIREKITE    instance-20240906-204153         +0 00:00:04 92k8p7q5hh6bf runquery@instan  SQL*Net more data to client     4454      0
1:1418    WIREKITE    instance-20240906-204153         +0 00:00:04 as5mwg3dp83wb runquery@instan  SQL*Net more data to client     4057      0
1:1347    WIREKITE    instance-20240906-204153         +0 00:00:04 fqpyzcy3qkytg runquery@instan  db file sequential read        4691      0
1:1299    WIREKITE    instance-20240906-204153         +0 00:00:05 0kyfxa2t2tp7v runquery@instan  db file sequential read        5522      0
1:1179    WIREKITE    instance-20240906-204153         +0 00:00:05 3xt6n4mxuprmv runquery@instan  db file sequential read        6242      0
1:1154    WIREKITE    instance-20240906-204153         +0 00:00:05 5uq2747pb6jk9 runquery@instan  ON CPU                6194      0
1:1275    WIREKITE    instance-20240906-204153         +0 00:00:05 65rqpvqsv76fq runquery@instan  db file sequential read        5731      0
1:1203    WIREKITE    instance-20240906-204153         +0 00:00:05 8bqtnz43039fh runquery@instan  SQL*Net more data to client     6045      0
1:985    WIREKITE    instance-20240906-204153         +0 00:00:05 ar773z9udt2pg runquery@instan  ON CPU                6326      0
1:1106    WIREKITE    instance-20240906-204153         +0 00:00:05 c14pkzp1w8x3n runquery@instan  SQL*Net more data to client     6164      0
1:1130    WIREKITE    instance-20240906-204153         +0 00:00:05 f1ph8q6xxp18g runquery@instan  db file sequential read        6212      0
1:1249    WIREKITE    instance-20240906-204153         +0 00:00:05 gfkuzggduvd7v runquery@instan  ON CPU                5244      0
1:1227    WIREKITE    instance-20240906-204153         +0 00:00:05 gz6vhhdtv3zhu runquery@instan  SQL*Net more data to client     6043      0
1:1034    WIREKITE    instance-20240906-204153         +0 00:00:06 0phgv66z4yr4u runquery@instan  ON CPU                7160      0
1:963    WIREKITE    instance-20240906-204153         +0 00:00:06 3bkn5h3ab3kzw runquery@instan  SQL*Net more data to client     7524      0
1:1059    WIREKITE    instance-20240906-204153         +0 00:00:06 8p1109f4puvf8 runquery@instan  ON CPU                6709      0
1:1010    WIREKITE    instance-20240906-204153         +0 00:00:06 fn1qdrs7m20h0 runquery@instan  ON CPU                7357      0
1:1083    WIREKITE    instance-20240906-204153         +0 00:00:06 gsjrhtyca5c75 runquery@instan  db file sequential read        6706      0
1:867    WIREKITE    instance-20240906-204153         +0 00:00:07 04pczjgn6nfc2 runquery@instan  ON CPU                8553      0
1:942    WIREKITE    instance-20240906-204153         +0 00:00:07 0x67wt8sz7pqv runquery@instan  db file sequential read        7847      0
1:460    WIREKITE    instance-20240906-204153         +0 00:00:07 3au7843wvzn1u runquery@instan  ON CPU                9084      0
1:913    WIREKITE    instance-20240906-204153         +0 00:00:07 4nrht8urvwgch runquery@instan  db file sequential read        7906      0
1:844    WIREKITE    instance-20240906-204153         +0 00:00:07 6gf43vy81uv0u runquery@instan  ON CPU                8522      0
1:820    WIREKITE    instance-20240906-204153         +0 00:00:07 8drjj5gnk9qqw runquery@instan  ON CPU                8459      0
1:890    WIREKITE    instance-20240906-204153         +0 00:00:07 8g1r2dpqnvdbr runquery@instan  SQL*Net more data to client     8134      0
1:293    WIREKITE    instance-20240906-204153         +0 00:00:08 56fb3yjcgcm0y runquery@instan  SQL*Net more data to client     9399      0
1:1515    WIREKITE    instance-20240906-204153         +0 00:00:08 aah6qs5vtyhuc runquery@instan  SQL*Net more data to client     9321      0
1:1490    WIREKITE    instance-20240906-204153         +0 00:00:08 fw6fc9zwk2dyq runquery@instan  db file sequential read        9696      0
1:1443    WIREKITE    instance-20240906-204153         +0 00:00:08 gth67y255by8r runquery@instan  db file sequential read        9774      0
1:1417    WIREKITE    instance-20240906-204153         +0 00:00:09 dy22d8j8npat0 runquery@instan  SQL*Net more data to client        10626      0
1:289    SYS        instance-20240906-204153         nul         nul       nul            *IDLE* OFS idle               10528      2
1:700    WIREKITE    instance-20240906-204153         nul         25yx2p6anpkty runquery@instan  ON CPU                  19      0

64 rows selected.

Wirekite Run 1

We had a standard 64 CPU GCP instance with 256 GB RAM. The host was Linux RHEL8 with Oracle version 19c. The machine lived in us-central1 region. We rebooted instances between runs.

For Snowflake we created a Large size warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We ran with 64 wirekite threads.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

And this was the result. It took about 11 minutes to move 1 billion rows from oracle to snowflake.

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Fri Mar 14 00:39:48 UTC 2025
FINISH: Fri Mar 14 00:50:24 UTC 2025
ELAPSED: 0 hrs 10 min 36 sec

Wirekite Run 2

We upgrade the oracle machine to 128 CPU, 512 GB RAM, but not changing any oracle parameters - keeping the same memory and buffer cache values.

We also upgraded the snowflake to X Large with Standard type.

We ran with 128 wirekite threads.

This is what we got. A whopping 6 minutes and 9 seconds to migrate a BILLION rows from Oracle to Snowflake.

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Fri Mar 14 00:13:06 UTC 2025
FINISH: Fri Mar 14 00:19:15 UTC 2025
ELAPSED: 0 hrs 6 min 9 sec

SQL Server to Snowflake - 10 million ops (inserts + updates + deletes) - 1 mins 45 secs

Wirekite — Wed, 04 Jun 2025 07:00:00 GMT

So we set up a Change Data Capture replication pipeline from SQL Server to Snowflake and wanted to benchmark how it performs under a “typical” OLTP load. This article discusses the various characteristics of the benchmark.

Benchmark Details

It has multiple tables.
It has some inserts.
It has somewhat more updates.
It has some deletes.
It has small transactions with lot of commits, mostly single row operations and commits.

Given the above characteristics we decided on doing the following for the benchmark.

We will have 10 tables.
We will have 2 million single row inserts.
We will have 6 million single row updates.
We will have 2 million single row deletes.
We will have auto commit on, causing all the above operations to have corresponding commit.

Table Structure

We will have 10 tables called firenibble 1 to 10.

CREATE TABLE dbo.firenibble1 (
    f0 bigint NOT NULL,
    f1 bigint DEFAULT NULL,
    f2 bigint DEFAULT NULL,
    f3 int DEFAULT NULL,
    f4 float DEFAULT NULL,
    f5 float DEFAULT NULL,
    f6 float DEFAULT NULL,
    f7 float DEFAULT NULL,
    f8 varchar(128) DEFAULT NULL,
    f9 varchar(128) DEFAULT NULL,
    f10 date DEFAULT NULL,
    f11 date DEFAULT NULL,
    f12 date DEFAULT NULL,
    f13 varchar(128) DEFAULT NULL,
    f14 varchar(128) DEFAULT NULL,
    f15 varchar(128) DEFAULT NULL,
    PRIMARY KEY (f0)
  );

Insert

Insert will essentially be new primary key insert.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Update

Update will be updating of two columns on a primary key value.

update firenibble1 set f2 = 280396830734761092, f9 = 'n6ChQjN9dfxIl6nc6SNgIzSahfeXfAef' where f0 = 19998;

Delete

Delete will be essentially a primary key delete.

delete from firenibble1 where f0 = 19998;

DML Operations Sequence

I know we could have been more granular mixing the inserts, updates and deletes even more, and as we do more benchmarks we might try that strategy but for starters this looked pretty good. Also we wanted to see how our replication performed with large number of insert only, update only and delete only loads. 1 million inserts

1 million inserts
3 million updates
1 million deletes
1 million inserts
3 million updates
1 million deletes.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble2 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble3 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble4 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble5 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble6 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble7 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble8 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble9 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble10 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Benchmark Sequence

A choice we had was to make was do one of the following

Have the wirekite replication running. Run the load and wait for the last event of the load to appear on the target. Measure the time between when the load started and the last event appeared.
Run the load and note down the binlog number and position at which the load started and wait for the load to finish. Now start the wirekite replication from the starting binlog number and position and wait for the last event of the load to appear on the target. Measure the time between when the wirekite replication started and the last event appeared on the target.

Machine Configuration

For SQL Server we created an n2-standard-16 GCP instance with all the standard configurations - 8 CPU’s and 64 GB RAM in the us-central region. We installed Ubuntu version 24 and SQL Server version 19.

For Snowflake we created a Large Size Warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

SQL Server Run Around

SQL Server had a really different change capturing scheme than all the other databases. While all the other databases use a central repository of some kinds to track changes (Oracle stores changes in archive logs, MySQL stores them in binlogs, Postgres uses WAL and so forth), and the changes are also stored serially across this data store. This means that the changes need to be traversed in order and you have really no opportunity for parallelism.

SQL Server took a very different route - they chose to have “change tables”. When you enable change tracking on a given table, a corresponding change table is created with all the columns of the original table as well as some meta columns to track the change order.

https://learn.microsoft.com/en-us/sql/relational-databases/system-tables/cdc-capture-instance-ct-transact-sql?view=sql-server-ver17

This means that one can actually have multiple threads tracking changes across multiple tables, and this is why this is one of our fastest benchmark.

Also note that SQL Server chose to use a cryptic LSN ( as opposed to humanly readable SCN in oracle and humanly readable binlog coordinate in MySQL). The fact that it was cryptic made it really difficult to see order in changes since you can’t tell which change happened before/after the other one.

The multi table approach also caused another problem - a transaction that spanned multiple tables appeared in multiple change tables. And SQL Server has no way of telling which tables are involved in a given transaction. So you have to scan all the change tables for a given LSN and stitch it together.

But one of the good things about SQL Server change tracking was that only committed transaction showed up in the change tables, unlike oracle where an uncommitted transaction can happily show up in an archive log, only to become committed in a later archive log. This was a relief.

No Airbyte Run

We could not do an airbyte run since it requires an enterprise level connector. So this is just our benchmark.

Wirekite Run

So how long did it take for wirekite to move 10 million changes — 1 minutes and 45 seconds.

cat wirekite_cdc.log | grep -E 'START|FINISH|ELAPSED'
START: Tue Jun 17 01:48:14 UTC 2025
FINISH: Tue Jun 17 01:49:59 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 1 min 45 sec

Oracle to Firebolt - 1B rows - 11 minutes 58 seconds

Wirekite — Tue, 03 Jun 2025 07:00:00 GMT

So after doing Firebolt benchmarks with Postgres and MySQL we went with Oracle. And as always we cranked it up to see how far does it go before crashing the Firebolt engine. And we came pretty close ! One thing I do want to mention here is that we would have probably gotten a better benchmark if we had an L sized Firebolt engine. But we are a startup and did not wanted to pay big bucks. So this benchmark is done with an M sized engine, even though things have been cranked up a lot on the Oracle side (128 CPU).

Also we could not run a benchmark with Airbyte since it is an enterprise license, and we did not want to go that route. Or to be putting it politely we couldn’t afford it.

Benchmark Details

Table

As always used our benchmark table. It has 15 columns. The columns are of very canonical datatypes - numbers of all shapes and sizes, medium sized strings and some dates. We wanted to make sure these datatypes can be transferred to most databases.

CREATE TABLE app_db.firenibble (
    f0    number(38)    NOT NULL,
    f1    number(38),
    f2    number(38),
    f3    int,
    f4    float,
    f5    float,
    f6    float,
    f7    float,
    f8    varchar2(128),
    f9    varchar2(128),
    f10    date,
    f11    date,
    f12    date,
    f13    varchar2(128),
    f14    varchar2(128),
    f15    varchar2(128),
    PRIMARY KEY (f0)
);

And here’s the 1 Billion rows that we inserted in Oracle.

SQL> select count(*) from app_db.firenibble;

  COUNT(*)
----------
1000000000

1 row selected.

SQL>

Oracle Stuff

We wanted to mention this a bit to talk about the uphill challenge that was getting oracle to be up and running and importing data into it.

We realized that some given environment variable has to be set to make sure the GUI runs as intended. Once we did that we got the right GUI and got Oracle installed.

And then came the challenge of configuring the database with sensible values so that you can actually load a reasonable amount of data in it. We had to configure

Configure and start the LISTENER. Understand LISTENER.ORA and TNSNAMES.ORA.
Figure out the differences between PFILE and SPFILE and the workings of SCOPE.
Turn off Automatic Shared Memory Management - MEMORY_TARGET = 0
Turn off Automatic Management - SGA_TARGET = 0
Set SGA_MAX_SIZE
Set DB_CACHE_SIZE
Set SHARED_POOL_SIZE
Set LARGE_POOL_SIZE
Modify a few kernel parameters (SHMMAX, SHMALL, file open limits), modify sysctl.conf and limits.conf and reboot the instance a few times.
Create a separate TABLESPACE to hold the table with the right number and size of data files and make sure they extend automatically.
Grant privileges on this TABLESPACE to the right USER.
Create the table with the right TABLESPACE for data but also the right TABLESPACE for INDEX otherwise it will create index in the default tablespace.
Figure out SQL LOADER to load the data from a csv into the table. Make sure you have the right commit frequency.
Monitor DBA_FREE_SPACE and add data files when it comes close.
Play with V$SESSION and V$SESSION_WAIT to see what is happening.
Play with EXPLAIN PLANS since Oracle gives a thousand ways to see the actual explain plan.

And phew !! You have an oracle database and the table with the data that you desire.

As you can see 128 rows - 128 CPU pushed to their max. Just how we like it.

INST_ID                                                   WAIT
:SID    USERNAME    MACHINE           SQL_ID      EVENT                  (SEC)
------- --------------- ------------------------- --------------- --------------------------------- ------
1:1095    WIREKITE    instance-20240906-204153  g7pakv22nf5k5   db file sequential read         0
1:781    WIREKITE    instance-20240906-204153  fp8p2f77qk3x3   ON CPU                 0
1:973    WIREKITE    instance-20240906-204153  38hztq2rqb641   SQL*Net more data to client         0
1:961    WIREKITE    instance-20240906-204153  5m4sdx6rwq95h   db file sequential read         0
1:1021    WIREKITE    instance-20240906-204153  76s40b7hwpxb0   SQL*Net more data to client         0
1:997    WIREKITE    instance-20240906-204153  bx88bvp7czy31   SQL*Net more data to client         0
1:1285    WIREKITE    instance-20240906-204153  0b9hp762bv399   SQL*Net more data to client         0
1:1083    WIREKITE    instance-20240906-204153  1fv0rnd0d72r5   SQL*Net more data to client         0
1:1153    WIREKITE    instance-20240906-204153  2tyxx90zuk1q2   SQL*Net more data to client         0
1:1237    WIREKITE    instance-20240906-204153  40hw44jqarbg4   ON CPU                 0
1:1178    WIREKITE    instance-20240906-204153  507771twhbkw4   SQL*Net more data to client         0
1:1333    WIREKITE    instance-20240906-204153  5bfubswhvwjn5   SQL*Net more data to client         0
1:1225    WIREKITE    instance-20240906-204153  6wj8brk2db946   db file sequential read         0
1:1273    WIREKITE    instance-20240906-204153  7gctuq46a8rpc   db file sequential read         0
1:1213    WIREKITE    instance-20240906-204153  81vd9na2vtagd   SQL*Net more data to client         0
1:1129    WIREKITE    instance-20240906-204153  99fz5udxxnt85   db file sequential read         0
1:1309    WIREKITE    instance-20240906-204153  9ttgr05j0g4m7   ON CPU                 0
1:1070    WIREKITE    instance-20240906-204153  az9fbzv7v3vuu   SQL*Net more data to client         0
1:1201    WIREKITE    instance-20240906-204153  azbwu204rvra1   db file sequential read         0
1:1046    WIREKITE    instance-20240906-204153  b73m35uvk887d   ON CPU                 0
1:1165    WIREKITE    instance-20240906-204153  bavddw44z91y3   db file sequential read         0
1:1249    WIREKITE    instance-20240906-204153  fsgbvc82873pp   db file sequential read         0
1:1297    WIREKITE    instance-20240906-204153  gp72k5ug0bzqh   SQL*Net more data to client         0
1:290    WIREKITE    instance-20240906-204153  01r6nk321c1mu   ON CPU                 0
1:1381    WIREKITE    instance-20240906-204153  158731r73vt9j   db file sequential read         0
1:1033    WIREKITE    instance-20240906-204153  15kd0wc9g60hb   db file sequential read         0
1:1119    WIREKITE    instance-20240906-204153  4gm01vu41pw4u   ON CPU                 0
1:1261    WIREKITE    instance-20240906-204153  6kf7k1gp9uahy   db file sequential read         0
1:171    WIREKITE    instance-20240906-204153  6npmrk46q0vdj   ON CPU                 0
1:1191    WIREKITE    instance-20240906-204153  8hzk0chgnuqca   db file sequential read         0
1:1345    WIREKITE    instance-20240906-204153  9rrhpfu34wvgg   ON CPU                 0
1:158    WIREKITE    instance-20240906-204153  a4su8wmz91zdk   ON CPU                 0
1:1501    WIREKITE    instance-20240906-204153  dpbnwgk8bm630   SQL*Net more data to client         0
1:75    WIREKITE    instance-20240906-204153  08smhdfa4r4np   ON CPU                 0
1:1    WIREKITE    instance-20240906-204153  0k2au740xs4cy   ON CPU                 0
1:13    WIREKITE    instance-20240906-204153  0mgwahtaumhk3   ON CPU                 0
1:86    WIREKITE    instance-20240906-204153  18bz44xnhfdn6   ON CPU                 0
1:218    WIREKITE    instance-20240906-204153  27h9njd2x453w   ON CPU                 0
1:1393    WIREKITE    instance-20240906-204153  2p2t8du5r4q5b   db file sequential read         0
1:1441    WIREKITE    instance-20240906-204153  2xkr7c5urs9av   db file sequential read         0
1:302    WIREKITE    instance-20240906-204153  315rp762ufvf2   SQL*Net more data to client         0
1:50    WIREKITE    instance-20240906-204153  356628aj189y6   ON CPU                 0
1:1369    WIREKITE    instance-20240906-204153  4q7pqa4awvg13   SQL*Net more data to client         0
1:1357    WIREKITE    instance-20240906-204153  4u08sk6rq38sg   SQL*Net more data to client         0
1:62    WIREKITE    instance-20240906-204153  67mbpqqzz2cvj   ON CPU                 0
1:194    WIREKITE    instance-20240906-204153  6mxdbwnkgg3za   ON CPU                 0
1:134    WIREKITE    instance-20240906-204153  6ud8n88tr1z73   ON CPU                 0
1:278    WIREKITE    instance-20240906-204153  6vr6374cyb99t   db file sequential read         0
1:111    WIREKITE    instance-20240906-204153  6w9102strfqr0   SQL*Net more data to client         0
1:230    WIREKITE    instance-20240906-204153  7ucatg27a5whu   ON CPU                 0
1:1321    WIREKITE    instance-20240906-204153  850fj4fk4ctj6   SQL*Net more data to client         0
1:26    WIREKITE    instance-20240906-204153  bhhwkm7an9w4x   SQL*Net more data to client         0
1:1489    WIREKITE    instance-20240906-204153  cwvpn8r7jmsj4   db file sequential read         0
1:1513    WIREKITE    instance-20240906-204153  fr8xuvzn69n3d   SQL*Net more data to client         0
1:1429    WIREKITE    instance-20240906-204153  2mxxtnudav2jh   SQL*Net more data to client         0
1:531    WIREKITE    instance-20240906-204153  2xdb3g1uj8j76   db file sequential read         0
1:542    WIREKITE    instance-20240906-204153  2xuacbbh7nrtq   ON CPU                 0
1:434    WIREKITE    instance-20240906-204153  39darj310bbpx   ON CPU                 0
1:458    WIREKITE    instance-20240906-204153  3vhftrap7gt7u   ON CPU                 0
1:1465    WIREKITE    instance-20240906-204153  4t492w9zpt8w9   db file sequential read         0
1:410    WIREKITE    instance-20240906-204153  5c3pkzwd61pgm   ON CPU                 0
1:206    WIREKITE    instance-20240906-204153  5hwrfgnwvw59g   ON CPU                 0
1:1417    WIREKITE    instance-20240906-204153  6rhvsb3hxakth   SQL*Net more data to client         0
1:506    WIREKITE    instance-20240906-204153  89tmwx6m7vmpd   ON CPU                 0
1:314    WIREKITE    instance-20240906-204153  959dftbyd3akw   ON CPU                 0
1:326    WIREKITE    instance-20240906-204153  bf4069bm6ac6z   ON CPU                 0
1:494    WIREKITE    instance-20240906-204153  chtd5700kg1y0   ON CPU                 0
1:422    WIREKITE    instance-20240906-204153  d3xv12cs8cpx6   db file sequential read         0
1:374    WIREKITE    instance-20240906-204153  d4amkamq2adv4   SQL*Net more data to client         0
1:446    WIREKITE    instance-20240906-204153  dcsj735hkfwuz   db file sequential read         0
1:242    WIREKITE    instance-20240906-204153  djba5d36155fg   ON CPU                 0
1:254    WIREKITE    instance-20240906-204153  duffm4fk4t0v3   ON CPU                 0
1:338    WIREKITE    instance-20240906-204153  fw0y58p1j9yd8   SQL*Net more data to client         0
1:519    WIREKITE    instance-20240906-204153  gfqzbjtkp73yf   SQL*Net more data to client         0
1:99    WIREKITE    instance-20240906-204153  gpasjhbh4fa8j   ON CPU                 0
1:577    WIREKITE    instance-20240906-204153  015p1b3nbkqvc   ON CPU                 0
1:661    WIREKITE    instance-20240906-204153  0975yasnaj84s   ON CPU                 0
1:398    WIREKITE    instance-20240906-204153  0ynvbuwbfdw07   ON CPU                 0
1:673    WIREKITE    instance-20240906-204153  1ska47yqfpcvk   db file sequential read         0
1:482    WIREKITE    instance-20240906-204153  28gm4phpbf4ac   ON CPU                 0
1:685    WIREKITE    instance-20240906-204153  2gmjbw3jqk679   SQL*Net more data to client         0
1:266    WIREKITE    instance-20240906-204153  3jsdn4ug83781   ON CPU                 0
1:613    WIREKITE    instance-20240906-204153  4y36j319p82b7   SQL*Net more data to client         0
1:735    WIREKITE    instance-20240906-204153  51cvp6bqgz8x2   db file sequential read         0
1:350    WIREKITE    instance-20240906-204153  6afq0f90fr3bm   ON CPU                 0
1:567    WIREKITE    instance-20240906-204153  70qwru3066dmd   db file sequential read         0
1:747    WIREKITE    instance-20240906-204153  84dc8hjzvzp7r   SQL*Net more data to client         0
1:721    WIREKITE    instance-20240906-204153  aa0xj74hkvaga   SQL*Net more data to client         0
1:651    WIREKITE    instance-20240906-204153  aurk5kvvurn0p   SQL*Net more data to client         0
1:591    WIREKITE    instance-20240906-204153  bcyx6agc775c0   db file sequential read         0
1:806    WIREKITE    instance-20240906-204153  c2n80x6mnhxug   ON CPU                 0
1:1453    WIREKITE    instance-20240906-204153  c7q8qmck82c0t   db file sequential read         0
1:147    WIREKITE    instance-20240906-204153  cc13fpt8x6333   db file sequential read         0
1:1143    WIREKITE    instance-20240906-204153  cck3huajnx049   ON CPU                 0
1:603    WIREKITE    instance-20240906-204153  chvpbd00683mn   ON CPU                 0
1:1477    WIREKITE    instance-20240906-204153  ck08wccrdj779   db file sequential read         0
1:697    WIREKITE    instance-20240906-204153  dcfqd69fu43r6   SQL*Net more data to client         0
1:637    WIREKITE    instance-20240906-204153  fqu65yb2z41g4   ON CPU                 0
1:709    WIREKITE    instance-20240906-204153  gc1khr7cb0jkr   db file sequential read         0
1:771    WIREKITE    instance-20240906-204153  0bbhpg4xh1quz   SQL*Net more data to client         0
1:914    WIREKITE    instance-20240906-204153  0p963knq6ug5k   ON CPU                 0
1:962    WIREKITE    instance-20240906-204153  0yfrquyhguvkk   SQL*Net more data to client         0
1:842    WIREKITE    instance-20240906-204153  2csqptc251j6m   db file sequential read         0
1:855    WIREKITE    instance-20240906-204153  2qy7dp5h6j683   db file sequential read         0
1:903    WIREKITE    instance-20240906-204153  5zttc28j8an72   SQL*Net more data to client         0
1:626    WIREKITE    instance-20240906-204153  6488r8g08pg98   ON CPU                 0
1:867    WIREKITE    instance-20240906-204153  6czbxsbpt40pa   SQL*Net more data to client         0
1:782    WIREKITE    instance-20240906-204153  9rx1amu5t7687   db file sequential read         0
1:891    WIREKITE    instance-20240906-204153  ar1u44uu4duc8   SQL*Net more data to client         0
1:819    WIREKITE    instance-20240906-204153  atb6tt99g545d   db file sequential read         0
1:759    WIREKITE    instance-20240906-204153  dtvncwjyn7q5w   SQL*Net more data to client         0
1:795    WIREKITE    instance-20240906-204153  du8vgs1fa71dc   SQL*Net more data to client         0
1:879    WIREKITE    instance-20240906-204153  f5mx7ufxd69nb   ON CPU                 0
1:831    WIREKITE    instance-20240906-204153  fs4fmhy0m526q   db file sequential read         0
1:122    WIREKITE    instance-20240906-204153  g8pjrt41acun3   ON CPU                 0
1:986    WIREKITE    instance-20240906-204153  0860mtbbqascq   db file sequential read         0
1:974    WIREKITE    instance-20240906-204153  7npm3f6qsgv96   SQL*Net more data to client         0
1:470    WIREKITE    instance-20240906-204153  7s0ksx9hw8y60   ON CPU                 0
1:927    WIREKITE    instance-20240906-204153  f4g4puxbhnjz3   SQL*Net more data to client         0
1:1358    WIREKITE    instance-20240906-204153  1njp9bm8ndt0s   SQL*Net more data to client         0
1:1346    WIREKITE    instance-20240906-204153  6bg4w57gv8h98   SQL*Net more data to client         0
1:1130    WIREKITE    instance-20240906-204153  70m7j9dgx8fdk   db file sequential read         0
1:1202    WIREKITE    instance-20240906-204153  89phb3p3vxtbx   ON CPU                 0
1:1226    WIREKITE    instance-20240906-204153  g5pp3j89hxnky   SQL*Net more data to client         0
1:1177    WIREKITE    instance-20240906-204153  gt1gg8k4ukpad   SQL*Net more data to client         0
1:1154    WIREKITE    instance-20240906-204153  3sukb15sm2fyp   SQL*Net more data to client         0
1:1370    WIREKITE    instance-20240906-204153  dwcphffv8vn15   db file sequential read         0
1:145    SYS        instance-20240906-204153  nul          *IDLE* OFS idle             3

128 rows selected.

SQL>

Firebolt Stuff

Firebolt has been a very interesting ride. They have decided on some very serious Engineering Tradeoffs. Their focus is just speed. And anything that comes in the way needs to go !!

We like it.

But they did not have a Command Line Utility. Which is a bit strange for a Database company focussed on creating a gnarly engineering product.

So we build our own Command Line Utility !!

=> ALTER ENGINE my_engine SET TYPE = "M" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 FAMILY=CO;
ok
=> show engines;
1: engine_name: my_engine
1: engine_owner: sandhu.nurat
1: type: M
1: family: COMPUTE_OPTIMIZED
1: nodes: 5
1: clusters: 1
1: status: RUNNING
1: auto_start: 1
1: auto_stop: 20
1: initially_stopped: 0
1: url: account-1-ewafh.api.us-east-1.app.firebolt.io?engine=my_engine
1: default_database:
1: version: 4.16.8
1: last_started: 2025-03-16 20:29:35.366
1: last_stopped: 2025-03-16 20:20:19.808
1: description:
1: fbu_rate: 40
=>

You can use it to check tables and various properties.

=> show tables;
1: table_name: firenibble
1: table_type: BASE TABLE
1: column_count: 16
1: primary_index: f0
1: schema: CREATE TABLE "firenibble" ("f0" BIGINT NOT NULL, "f1" BIGINT NULL, "f2" BIGINT NULL, "f3" INTEGER NULL, "f4" DOUBLE PRECISION NULL, "f5" DOUBLE PRECISION NULL, "f6" DOUBLE PRECISION NULL, "f7" DOUBLE PRECISION NULL, "f8" TEXT NULL, "f9" TEXT NULL, "f10" DATE NULL, "f11" DATE NULL, "f12" DATE NULL, "f13" TEXT NULL, "f14" TEXT NULL, "f15" TEXT NULL) PRIMARY INDEX "f0"
1: number_of_rows: 1000000000
1: compressed_bytes: 216.93 GiB
1: uncompressed_bytes: 234.69 GiB
1: compression_ratio: 1
1: number_of_tablets: 1586
=>

Machine Configuration

For Firebolt we created an engine with M size (medium), 5 nodes, 2 clusters and it was COMPUTE_OPTIMIZED. We created the engine in us-east region. Firebolt does not offer us-central region. Also Firebolt does not offer L size by default that other cloud vendors offer. We got the following message when we tried to upgrade our engine.

 => ALTER ENGINE my_engine SET TYPE = "L" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2;
2025/03/15 22:46:31 ERROR query ALTER ENGINE my_engine SET TYPE = "L" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 failed: error during query execution: error during query request: Alter engine failed. 'L' node type is currently in preview. Reach out to Firebolt Support to enable 'L' node type for this account
=>

Wirekite Run 1

We had a standard 64 CPU GCP instance with 256 GB RAM. The host was Linux RHEL8 with Oracle version 19c. The machine lived in us-central1 region. We rebooted instances between runs.

We ran with 64 wirekite threads.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

And this was the result. It took about 16 minutes to move 1 billion rows from oracle to firebolt.

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sun Mar 16 02:36:51 UTC 2025
FINISH: Sun Mar 16 02:53:15 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 16 min 24 sec

Wirekite Run 2

We upped the source oracle instance to 128 CPU with 512 GB RAM. Rest everything remains same. This also forces an Oracle instance start and makes the buffer cold.

We ran with 128 threads.

We migrated 1 BILLION rows in 11 minutes and 58 seconds.

$ cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sun Mar 16 03:50:35 UTC 2025
FINISH: Sun Mar 16 04:02:33 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 11 min 58 sec

And here’s the 1 Billion rows.

INFO[0000]log.go:182 gofirebolt.(*defaultLogger).Infof Credentials cache path: /home/ziggy/.cache/firebolt/temporary_credential.json
firebolt instance connect successful
=> select count(*) as count from firenibble;
1: count: 1000000000

And on the GUI

SQL Server to Firebolt - 10 million ops (inserts + updates + deletes) - 1 mins 13 secs

Wirekite — Tue, 03 Jun 2025 07:00:00 GMT

So we set up a Change Data Capture replication pipeline from SQL Server to Firebolt and wanted to benchmark how it performs under a “typical” OLTP load. This article discusses the various characteristics of the benchmark.

Benchmark Details

It has multiple tables.
It has some inserts.
It has somewhat more updates.
It has some deletes.
It has small transactions with lot of commits, mostly single row operations and commits.

Given the above characteristics we decided on doing the following for the benchmark.

We will have 10 tables.
We will have 2 million single row inserts.
We will have 6 million single row updates.
We will have 2 million single row deletes.
We will have auto commit on, causing all the above operations to have corresponding commit.

Table Structure

We will have 10 tables called firenibble 1 to 10.

CREATE TABLE dbo.firenibble1 (
    f0 bigint NOT NULL,
    f1 bigint DEFAULT NULL,
    f2 bigint DEFAULT NULL,
    f3 int DEFAULT NULL,
    f4 float DEFAULT NULL,
    f5 float DEFAULT NULL,
    f6 float DEFAULT NULL,
    f7 float DEFAULT NULL,
    f8 varchar(128) DEFAULT NULL,
    f9 varchar(128) DEFAULT NULL,
    f10 date DEFAULT NULL,
    f11 date DEFAULT NULL,
    f12 date DEFAULT NULL,
    f13 varchar(128) DEFAULT NULL,
    f14 varchar(128) DEFAULT NULL,
    f15 varchar(128) DEFAULT NULL,
    PRIMARY KEY (f0)
  );

Insert

Insert will essentially be new primary key insert.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Update

Update will be updating of two columns on a primary key value.

update firenibble1 set f2 = 280396830734761092, f9 = 'n6ChQjN9dfxIl6nc6SNgIzSahfeXfAef' where f0 = 19998;

Delete

Delete will be essentially a primary key delete.

delete from firenibble1 where f0 = 19998;

DML Operations Sequence

1 million inserts
3 million updates
1 million deletes
1 million inserts
3 million updates
1 million deletes.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble2 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble3 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble4 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble5 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble6 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble7 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble8 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble9 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble10 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Benchmark Sequence

A choice we had was to make was do one of the following

Have the wirekite replication running. Run the load and wait for the last event of the load to appear on the target. Measure the time between when the load started and the last event appeared.
Run the load and note down the binlog number and position at which the load started and wait for the load to finish. Now start the wirekite replication from the starting binlog number and position and wait for the last event of the load to appear on the target. Measure the time between when the wirekite replication started and the last event appeared on the target.

Machine Configuration

For Firebolt we created a Medium Size Warehouse also in the us-central1 region to minimize the network distance.

=> ALTER ENGINE my_engine SET TYPE = "M" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 FAMILY=CO;
ok

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

SQL Server Run Around

https://learn.microsoft.com/en-us/sql/relational-databases/system-tables/cdc-capture-instance-ct-transact-sql?view=sql-server-ver17

This means that one can actually have multiple threads tracking changes across multiple tables, and this is why this is one of our fastest benchmark.

No Airbyte Run

We could not do an airbyte run since it requires an enterprise level connector. So this is just our benchmark.

Wirekite Run

So how long did it take for wirekite to move 10 million changes — 1 minutes and 13 seconds.

cat wirekite_cdc.log | grep -E 'START|FINISH|ELAPSED'
START: Fri Jun 13 05:07:24 UTC 2025
FINISH: Fri Jun 13 05:08:37 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 1 min 13 sec

PostgreSQL to Snowflake - 10 million ops (inserts + updates + deletes) - 3 mins 52 secs

Wirekite — Mon, 02 Jun 2025 07:00:00 GMT

So we set up a Change Data Capture replication pipeline from PostgreSQL to Snowflake and wanted to benchmark how it performs under a “typical” OLTP load. This article discusses the various characteristics of the benchmark.

Benchmark Details

It has multiple tables.
It has some inserts.
It has somewhat more updates.
It has some deletes.
It has small transactions with lot of commits, mostly single row operations and commits.

Given the above characteristics we decided on doing the following for the benchmark.

We will have 10 tables.
We will have 2 million single row inserts.
We will have 6 million single row updates.
We will have 2 million single row deletes.
We will have auto commit on, causing all the above operations to have corresponding commit.

Table Structure

We will have 10 tables called firenibble 1 to 10.

CREATE TABLE public.firenibble1 (
  f0 bigint NOT NULL,
  f1 bigint DEFAULT NULL,
  f2 bigint DEFAULT NULL,
  f3 int DEFAULT NULL,
  f4 float8 DEFAULT NULL,
  f5 float8 DEFAULT NULL,
  f6 float8 DEFAULT NULL,
  f7 float8 DEFAULT NULL,
  f8 varchar(128) DEFAULT NULL,
  f9 varchar(128) DEFAULT NULL,
  f10 date DEFAULT NULL,
  f11 date DEFAULT NULL,
  f12 date DEFAULT NULL,
  f13 varchar(128) DEFAULT NULL,
  f14 varchar(128) DEFAULT NULL,
  f15 varchar(128) DEFAULT NULL,
  PRIMARY KEY (f0)
) ;

Insert

Insert will essentially be new primary key insert.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Update

Update will be updating of two columns on a primary key value.

update firenibble1 set f2 = 280396830734761092, f9 = 'n6ChQjN9dfxIl6nc6SNgIzSahfeXfAef' where f0 = 19998;

Delete

Delete will be essentially a primary key delete.

delete from firenibble1 where f0 = 19998;

DML Operations Sequence

Then came the question of how do we order the load so that we can rerun the load over and over again and never have to clean up the previous load. So we decided that the load will be such that we will start with 0 rows in all 10 tables, do a bunch of inserts, do a bunch of updates, do a bunch of deletes and end up with 0 rows in all tables. This kept the load and the tables rather clean. So the exact sequence will look like this. I know we could have been more granular mixing the inserts, updates and deletes even more, and as we do more benchmarks we might try that strategy but for starters this looked pretty good. Also we wanted to see how our replication performed with large number of insert only, update only and delete only loads. Also note that there is no relationships between the load and the change logs, in the sense there is no flush binary logs happening anywhere causing the inserts, updates and deletes to cross binary log boundaries.

1 million inserts
3 million updates
1 million deletes
1 million inserts
3 million updates
1 million deletes.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble2 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble3 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble4 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble5 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble6 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble7 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble8 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble9 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble10 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Benchmark Sequence

A choice we had was to make was do one of the following

Have the wirekite replication running. Run the load and wait for the last event of the load to appear on the target. Measure the time between when the load started and the last event appeared.
Run the load and note down the binlog number and position at which the load started and wait for the load to finish. Now start the wirekite replication from the starting binlog number and position and wait for the last event of the load to appear on the target. Measure the time between when the wirekite replication started and the last event appeared on the target.

Machine Configuration

For Postgres we created an n2-standard-16 GCP instance with all the standard configurations - 8 CPU’s and 64 GB RAM in the us-central region. We installed Ubuntu version 24 and Oracle version 19. Our archive log size was 330 MB.

For Snowflake we created a Large Size Warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

Wirekite Run

So how long did it take for wirekite to move 10 million changes — 3 minutes and 52 seconds.

cat wirekite_cdc.log | grep -E 'START|FINISH|ELAPSED'
START: Tue May 13 06:46:30 UTC 2025
FINISH: Tue May 13 06:50:22 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 4 min 5 sec

SQL Server to Snowflake - 1B rows - 10 minutes 52 seconds

Wirekite — Mon, 02 Jun 2025 07:00:00 GMT

So one more benchmark except this time with a really expensive enterprise grade database.

We could not compare this benchmark with Airbyte since Airbyte does not allow an open source connector for SQL Server. Entirely understandable. When we tried to configure the connector after spending a long time installing Airbyte we got the following message.

Benchmark Details

As usual we will use our firenibble table with 1 Billion rows. That’s a lot of rows.

Since we could not do an Airbyte benchmark we decided to play a little bit with SQL Server and see how far we can scale our benchmarks as we increased the source SQL Server machine configuration, the target Snowflake configurations and the number of wirekite threads.

Table

As usual we used our standard firenibble table.

CREATE TABLE dbo.firenibble (
    f0 bigint NOT NULL,
    f1 bigint DEFAULT NULL,
    f2 bigint DEFAULT NULL,
    f3 int DEFAULT NULL,
    f4 float DEFAULT NULL,
    f5 float DEFAULT NULL,
    f6 float DEFAULT NULL,
    f7 float DEFAULT NULL,
    f8 varchar(128) DEFAULT NULL,
    f9 varchar(128) DEFAULT NULL,
    f10 date DEFAULT NULL,
    f11 date DEFAULT NULL,
    f12 date DEFAULT NULL,
    f13 varchar(128) DEFAULT NULL,
    f14 varchar(128) DEFAULT NULL,
    f15 varchar(128) DEFAULT NULL,
    PRIMARY KEY (f0)
  );

The table also has 1 Billion rows.

=> select count(*) from dbo.firenibble1;
1: : 1000000000
=>

Loading Data in SQL Server

We used the following command to load the 1 Billion rows in the firenibble table. It took sometime but it worked.

BULK INSERT dbo.firenibble FROM '/mnt/spinning/csv/firenibble.csv' WITH (FIELDTERMINATOR = '\t', ROWTERMINATOR = '\n', FIRSTROW = 1, BATCHSIZE=5000);

Wirekite Run 1

We had a standard 128 CPU GCP instance with 512 GB RAM. The host was Linux Ubuntu 20.04 with SQL Server version 2022. The machine lived in us-central1 region. We rebooted instances between runs.

For Snowflake we created a Large size warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We ran with 96 wirekite threads.

And it took about 10 minutes and 55 seconds.

cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Mon Jun 16 23:58:11 UTC 2025
FINISH: Tue Jun 17 00:09:06 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 10 min 55 sec

Wirekite Run 2

Ran with same machine configurations but with 90 threads and got this - 10 minutes and 58 seconds. Oops the time increased.

cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Tue Jun 17 00:11:11 UTC 2025
FINISH: Tue Jun 17 00:22:09 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 10 min 58 sec

Wirekite Run 3

This time we ran with 84 threads and got our best time.

Moved 1 billion rows from SQL Server to Snowflake in 10 minutes and 52 seconds.

cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Mon Jun 16 23:44:02 UTC 2025
FINISH: Mon Jun 16 23:54:54 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 10 min 52 sec

I think there is still scope for improvement.

SQL Server to Firebolt - 1B rows - 12 minutes 19 seconds

Wirekite — Sun, 01 Jun 2025 07:00:00 GMT

So one more benchmark except this time with a really expensive enterprise grade database.

Benchmark Details

As usual we will use our firenibble table with 1 Billion rows. That’s a lot of rows.

Table

As usual we used our standard firenibble table.

CREATE TABLE dbo.firenibble (
    f0 bigint NOT NULL,
    f1 bigint DEFAULT NULL,
    f2 bigint DEFAULT NULL,
    f3 int DEFAULT NULL,
    f4 float DEFAULT NULL,
    f5 float DEFAULT NULL,
    f6 float DEFAULT NULL,
    f7 float DEFAULT NULL,
    f8 varchar(128) DEFAULT NULL,
    f9 varchar(128) DEFAULT NULL,
    f10 date DEFAULT NULL,
    f11 date DEFAULT NULL,
    f12 date DEFAULT NULL,
    f13 varchar(128) DEFAULT NULL,
    f14 varchar(128) DEFAULT NULL,
    f15 varchar(128) DEFAULT NULL,
    PRIMARY KEY (f0)
  );

The table also has 1 Billion rows.

=> select count(*) from dbo.firenibble1;
1: : 1000000000
=>

Loading Data in SQL Server

We used the following command to load the 1 Billion rows in the firenibble table. It took sometime but it worked.

BULK INSERT dbo.firenibble FROM '/mnt/spinning/csv/firenibble.csv' WITH (FIELDTERMINATOR = '\t', ROWTERMINATOR = '\n', FIRSTROW = 1, BATCHSIZE=5000);

Wirekite Run 1

We had a standard 128 CPU GCP instance with 512 GB RAM. The host was Linux Ubuntu 20.04 with SQL Server version 2022. The machine lived in us-central1 region. We rebooted instances between runs.

For Firebolt we created a Medium Size Warehouse also in the us-central1 region to minimize the network distance.

=> ALTER ENGINE my_engine SET TYPE = "M" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 FAMILY=CO;
ok

We ran with 84 wirekite threads.

And it took about 12 minutes and 35 seconds.

cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sun Jun 15 05:39:39 UTC 2025
FINISH: Sun Jun 15 05:52:14 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 12 min 35 sec

Wirekite Run 2

This time we ran with 96 threads and got our best time.

Moved 1 billion rows from SQL Server to Firebolt in 12 minutes and 19 seconds.

cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Sun Jun 15 05:39:39 UTC 2025
FINISH: Sun Jun 15 05:52:14 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 12 min 35 sec

PostgreSQL to Firebolt - 10 million ops (inserts + updates + deletes) - 2 mins 18 secs

Wirekite — Sun, 01 Jun 2025 07:00:00 GMT

So we set up a Change Data Capture replication pipeline from PostgreSQL to Firebolt and wanted to benchmark how it performs under a “typical” OLTP load. This article discusses the various characteristics of the benchmark.

Benchmark Details

It has multiple tables.
It has some inserts.
It has somewhat more updates.
It has some deletes.
It has small transactions with lot of commits, mostly single row operations and commits.

Given the above characteristics we decided on doing the following for the benchmark.

We will have 10 tables.
We will have 2 million single row inserts.
We will have 6 million single row updates.
We will have 2 million single row deletes.
We will have auto commit on, causing all the above operations to have corresponding commit.

Table Structure

We will have 10 tables called firenibble 1 to 10.

CREATE TABLE public.firenibble1 (
  f0 bigint NOT NULL,
  f1 bigint DEFAULT NULL,
  f2 bigint DEFAULT NULL,
  f3 int DEFAULT NULL,
  f4 float8 DEFAULT NULL,
  f5 float8 DEFAULT NULL,
  f6 float8 DEFAULT NULL,
  f7 float8 DEFAULT NULL,
  f8 varchar(128) DEFAULT NULL,
  f9 varchar(128) DEFAULT NULL,
  f10 date DEFAULT NULL,
  f11 date DEFAULT NULL,
  f12 date DEFAULT NULL,
  f13 varchar(128) DEFAULT NULL,
  f14 varchar(128) DEFAULT NULL,
  f15 varchar(128) DEFAULT NULL,
  PRIMARY KEY (f0)
) ;

Insert

Insert will essentially be new primary key insert.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Update

Update will be updating of two columns on a primary key value.

update firenibble1 set f2 = 280396830734761092, f9 = 'n6ChQjN9dfxIl6nc6SNgIzSahfeXfAef' where f0 = 19998;

Delete

Delete will be essentially a primary key delete.

delete from firenibble1 where f0 = 19998;

DML Operations Sequence

Then came the question of how do we order the load so that we can rerun the load over and over again and never have to clean up the previous load. So we decided that the load will be such that we will start with 0 rows in all 10 tables, do a bunch of inserts, do a bunch of updates, do a bunch of deletes and end up with 0 rows in all tables. This kept the load and the tables rather clean. So the exact sequence will look like this. I know we could have been more granular mixing the inserts, updates and deletes even more, and as we do more benchmarks we might try that strategy but for starters this looked pretty good. Also we wanted to see how our replication performed with large number of insert only, update only and delete only loads. Also note that there is no relationships between the load and the change logs, in the sense there is no flush binary logs happening anywhere causing the inserts, updates and deletes to cross binary log boundaries.

1 million inserts
3 million updates
1 million deletes
1 million inserts
3 million updates
1 million deletes.

insert into firenibble1 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble2 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble3 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble4 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble5 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble6 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble7 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble8 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble9 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');
insert into firenibble10 values (19998, 5963887289447355291, 6644703987735879175, -2094978734, 0.51946, 0.480314, 0.528386, 0.247536, 'Y4Jt79YtKat99KRV8CioT7p6bUVdtsm6', 'hC8SKt3MgvoGkOYJIljgKjdEaoY68YJe', '5354-04-29', '5741-02-18', '9548-08-16', 'K3fiVGzTGeiIB2AfChQDXOAQ7ANZBxEN', 'SETnGb7dxOVQthQtFdH6lbZ6yqVVoXPU', 'fJMS3fPnIJBnsi0CBxkFc2fZDf5gUiSJ');

Benchmark Sequence

A choice we had was to make was do one of the following

Have the wirekite replication running. Run the load and wait for the last event of the load to appear on the target. Measure the time between when the load started and the last event appeared.
Run the load and note down the binlog number and position at which the load started and wait for the load to finish. Now start the wirekite replication from the starting binlog number and position and wait for the last event of the load to appear on the target. Measure the time between when the wirekite replication started and the last event appeared on the target.

Machine Configuration

For Firebolt we created a Medium Size Warehouse also in the us-central1 region to minimize the network distance.

=> ALTER ENGINE my_engine SET TYPE = "M" NODES = 5 MIN_CLUSTERS=1 MAX_CLUSTERS = 2 FAMILY=CO;
ok

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

Wirekite Run

So how long did it take for wirekite to move 10 million changes — 5 minutes and 15 seconds.

cat wirekite_cdc.log | grep -E 'START|FINISH|ELAPSED'
START: Sun Jun 15 00:19:03 UTC 2025
FINISH: Sun Jun 15 00:24:18 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 5 min 15 sec

Wirekite Run 2 with Lower Commit Frequency

We did one more benchmark just to see how commits affect the performance and in fact they do. We got the the same 10 million ops with 2 minutes and 18 seconds.

cat wirekite_cdc.log | grep -E 'START|FINISH|ELAPSED'
START: Sun Jun 15 00:53:32 UTC 2025
FINISH: Sun Jun 15 00:55:49 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 2 min 18 sec