CDC setup

A bridge with the CDC trigger streams changes out of the source database's own change log the moment they commit — no polling. Each engine captures changes a different way and each has prerequisites Syncle cannot always set up for you. This page lists them per engine, shows what Syncle provisions itself, and gives the limits for each.

The alternative for live syncing is a watch bridge, which polls the source on a cursor and works on every engine, including the two cases where CDC falls short: SQLite has no change log at all, and the Redis change feed is not durable. The trigger modes are compared on How bridges work.

Per-engine prerequisites

PostgreSQL

Syncle uses logical replication with the built-in pgoutput plugin, so there is no server extension to install. Two things must be true on the server: wal_level=logical, and the connection's role has REPLICATION (superusers pass too). Changing wal_level needs a server restart, which is the one step Syncle cannot automate — on managed Postgres, set it in your provider's parameter group and reboot.

wal_level = logical

Grant replication with ALTER ROLE your_user REPLICATION;. The rest is provisioned for you: each CDC bridge gets a publication scoped to its source table and a logical replication slot, named syncle_pub_<id> and syncle_slot_<id> (the bridge id with dashes removed). If you later point the bridge at a different table, the publication is updated to match. The slot stores the confirmed position and is only advanced after Syncle has persisted its own cursor, so a restart resumes exactly where it left off without skipping changes.

What the table needs

PostgreSQL identifies the row an UPDATE or DELETE touched by the table's replica identity — its primary key, unless you have set something else. A table with no primary key and no replica identity can only report inserts. Worse, publishing updates or deletes for such a table makes those statements fail in your database ("cannot update table … because it does not have a replica identity and publishes updates"). Syncle checks this before it creates anything on the source and refuses to start the bridge rather than break the application that owns the table. You have three ways forward:

• capture insert only — Syncle then publishes only inserts;
• add a primary key;
• have the table send whole rows: ALTER TABLE your_table REPLICA IDENTITY FULL;

The same rule decides what a delete can do. A delete message carries only the replica-identity columns, so a target keyed on any other column would be handed a delete with no key in it — which matches nothing and leaves the row behind for ever, without an error. A bridge that captures deletes into a target keyed on a column the delete does not carry is refused at start, with the three fixes spelled out: key the target on the source's key, stop capturing deletes, or switch the table to REPLICA IDENTITY FULL. The readiness panel shows which columns the table identifies rows by.

How particular changes are handled

Replication slots and the source's disk

A replication slot makes PostgreSQL keep every byte of WAL written since the slot's position — for as long as the slot exists, whether or not anything is reading it. A bridge that is paused or has failed keeps its slot so that it can resume without a gap, and the source keeps accumulating WAL for it. Left alone for long enough, that fills the source's disk, and a PostgreSQL with a full disk stops accepting writes. This is the one way a Syncle bridge can hurt the database it reads from, so it is watched:

• Every minute (SYNCLE_SLOT_CHECK_SECONDS) Syncle measures how much WAL each CDC bridge's slot is pinning. Past SYNCLE_SLOT_WARN_BYTES (1 GiB) the bridge's job view shows a warning with the amount and what to do, and the same line goes to the log. It is also available from GET /api/bridges/:id/source-hold.
• The real safety net is on the server. On PostgreSQL 13+, set max_slot_wal_keep_size (for example 10GB): past it the server invalidates the slot instead of filling the disk, and that protects you even while Syncle itself is switched off. The readiness check tells you when it is unlimited, which is the default.
• Optionally, SYNCLE_SLOT_MAX_BYTES makes Syncle do the same from its side: a bridge that is not running and pins more than that has its slot dropped. It is off by default, because it trades a gap in that bridge for the source staying up, and that is a decision for whoever runs the source. A running bridge is never touched — it is behind, not abandoned.

A slot is released as soon as it is no longer needed: when the bridge is deleted, when it is edited into a watch or replay bridge, and when it is pointed at another connection or database (the slot lives on the old server, where nothing else would ever look for it again). If the drop fails — the server is unreachable at that moment — it is recorded and retried every minute until it succeeds; GET /api/bridges/cdc/cleanups lists what is still outstanding, and the connection it has to go through cannot be deleted in the meantime. To remove one by hand: SELECT pg_drop_replication_slot('syncle_slot_…');

Each CDC bridge uses one slot and one WAL sender, and a server has a fixed number of both (max_replication_slots, max_wal_senders; 10 each by default). The readiness check counts them, and a bridge is not started on a server that has none left.

A bridge that is running and has caught up holds next to nothing, also when its own table is quiet and the rest of the database is not: PostgreSQL 15 and later send a subscriber nothing for transactions that touch no published table, only keepalives, and Syncle answers those with the position the server reports whenever nothing it has received is still undelivered — as PostgreSQL's own subscribers do. (Versions up to 1.3 answered with their last delivery, and a bridge on a quiet table made the server keep all the WAL the other tables wrote for as long as it ran.)

Many tables, one slot

Thirty tables as thirty bridges are thirty slots, thirty WAL senders and thirty decodings of the same WAL — on a server that allows ten of each by default. A bridge whose Replication slot is set to Shared (trigger.slot: "shared") reads instead through one slot per source connection and database, which every shared bridge there uses: one connection, one decoding, each change handed to the bridges whose table it belongs to. They stay ordinary bridges — their own filters, transforms, targets, dead letters, schedule of verifications — and Bridge many tables in the bridge list (POST /api/bridges/bulk) makes one per table in a single step.

• The slot is confirmed as far as the slowest member has got — also a member that is stopped. That is what lets a member that comes back find everything it missed; it is also why a member left stopped holds WAL for all of them. Each member's page shows what it still has to read, the alert and the SYNCLE_SLOT_MAX_BYTES guard work per member (the guard gives up that member's place, not the slot), and when a member starts or resumes the stream restarts from the slowest position: the others are sent what they have had again, and drop it.
• A member joins behind a barrier. Adding a table to a slot that already exists has no consistent starting point of its own: a transaction that changed the table before it was published and commits afterwards would be neither in the copy nor in the stream. So the join waits for the transactions that were open at that moment to end (SYNCLE_SHARED_SLOT_JOIN_WAIT_MS, default 60 s) and only then takes its position. A transaction that never ends fails the start with its pid, user and application name instead.
• One member's operations are never imposed on another's table. A publication that publishes updates makes PostgreSQL refuse UPDATEs on a table without a replica identity, whoever reads them. The shared slot has one publication per set of operations (syncle_sp_…_i, …_iud, …), all created before the slot, and a table is only ever in the ones its own bridges ask for. Do not drop them: PostgreSQL fails the whole stream for a publication that is missing as of the change it is decoding, which is also why they are never created later.
• Members advance together. The stream waits for each delivery, so a member with a slow destination slows the others; with SYNCLE_CDC_SPOOL=on reading and delivering are decoupled. A member that fails stops by itself; the rest carry on.
• Changing the setting on an existing bridge releases what it had at once, and the bridge follows from its new place (or copies again, if that is how it starts): a position in one slot means nothing in another. The timeline says so, and reconcile closes the gap.

The last member to be deleted takes the slot and its publications with it. To look from the server's side: SELECT slot_name, active, confirmed_flush_lsn FROM pg_replication_slots WHERE slot_name LIKE 'syncle_shared_%';

When a bridge's place in the log is gone

A bridge resumes from a position in the source's change log, and that position can stop existing: the slot was dropped (by hand, by SYNCLE_SLOT_MAX_BYTES) or invalidated by the server (max_slot_wal_keep_size); MySQL purged the binlog file; MongoDB's oplog rolled past the resume token. Whatever changed at the source between that position and now can no longer be read.

Syncle does not paper over that. The bridge stops (or refuses to start) and says why; starting it again asks you to confirm Continue from now — POST /api/bridges/:id/watch/start with { "fromNow": true }. The timeline records the point where the gap is, and a replay of the same bridge brings the destination back in line. Earlier versions quietly made a new slot (or restarted the change stream) at the current position and carried on, leaving a hole in the destination that nothing showed.

MySQL

Syncle connects as a replication client and decodes row events from the binary log. Four server settings matter, and on managed MySQL they usually mean a parameter-group change plus a reboot:

log_bin          = ON
binlog_format    = ROW
binlog_row_image = FULL
server_id        = 1   # any unique id

The connecting user needs replication grants (a user with ALL PRIVILEGES also passes):

GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO CURRENT_USER;

There is nothing to provision — the binlog already exists. The cursor is the binlog file and position, so MySQL CDC is durable and resumes exactly after a restart, even in the middle of a multi-row statement.

Three limits are worth knowing before you point a bridge at MySQL.

binlog_transaction_compression is not supported. MySQL 8.0.20 and later can wrap a transaction's row events inside a compressed payload event, and the binlog reader has no decoder for it — the rows inside are not seen. Leave it OFF on a source you stream from.

MariaDB is untested. Connections, replay and watch bridges go through mysql2 and work, but CDC reads the binlog with a client that targets MySQL and that path has not been verified against MariaDB. Treat MariaDB CDC as unsupported until it has been.

Binlog positions belong to one server. A cursor records the server's @@server_uuid (and the GTID of the transaction it sits at). If the connection later reaches a different server — after a failover, say — the bridge refuses to resume rather than reading unrelated offsets, and says so.

MySQL purges its binlog on its own schedule (binlog_expire_logs_seconds, 30 days by default, often far less on managed servers), whoever still needs it. A bridge paused for longer than that has lost its place: the file its position is in no longer exists. Syncle checks for this before it starts a bridge, and in both cases asks you to confirm continuing from the current position — see when a bridge's place in the log is gone. Keep the binlog for at least as long as you might leave a bridge stopped.

MongoDB

Syncle opens a change stream on the source collection. Change streams require a replica set or a sharded cluster — they are not available on a standalone mongod. A single-node replica set is fine for development: start the server with --replSet rs0 and run rs.initiate() once. Managed MongoDB (Atlas) already satisfies this.

On MongoDB 6.0 and newer, Syncle enables change-stream pre-images on the source collection so a delete event carries the full prior document — without them a delete event contains only _id, and a bridge keyed on a business column could not find the row to remove downstream. On older servers this is a best-effort no-op and deletes carry only _id. The resume token is durable as long as it stays inside the oplog window. If the bridge is paused long enough for the oplog to roll past it, the bridge stops and says so, and starting it again asks you to confirm continuing from now — see when a bridge's place in the log is gone. Size the oplog for the longest pause you expect.

Redis

Syncle subscribes to keyspace notifications — pub/sub on the __keyevent@<db>__ channels of the connection's database index. The server must have notify-keyspace-events enabled with the E flag plus event classes. Syncle attempts CONFIG SET notify-keyspace-events EA itself when a bridge goes live; managed Redis may require enabling it in the provider console, and the readiness check tells you where you stand.

A notification carries only the key, so Syncle reads the current value afterwards, best-effort, and delivers rows shaped like { key, event, type, value } — deletes carry only key and event, because the value is already gone. Redis cannot tell a create from an overwrite, so every write is delivered as an update; del, unlink, expired and evicted arrive as deletes. Setting a TTL is not a delete — only the TTL actually firing is. A filter on the key column is a Redis-style glob, read the same way by the change stream, by a replay and by the copy a bridge makes before following: equals is the glob as written (user:*, session:??, h[ae]llo); contains, starts with and ends with are *value*, value* and *value. Syncle's own keys — its job queues, a bridge's spool — are never delivered, from a stream or a replay, when the Redis being read is the one Syncle runs on.

A Redis bridge that copies its keys first has no log to take a place in. It subscribes first and holds what it hears — the newest change per key — until the keys have been read (by following SCAN to its end, which returns every key that exists for the length of the read), then delivers what it held. If the process stops mid-copy, what was held is gone with it, like anything else Redis publishes to a subscriber that is not there.

A keyspace subscription hears every write to the database, including ones Syncle makes itself. Two consequences are handled for you. If the Redis you bridge from is also the Redis Syncle runs on (REDIS_URL — a single shared Redis is a common small setup), Syncle's own keys — its job queues under bull:bridge-jobs: and bull:bridge-watch:, and the CDC spool under syncle:cdc:spool: — are never treated as changes. And a bridge whose destination is the same Redis database it listens to is refused when it starts: every row it wrote would be captured as a new change and written again, without end. A different database number on the same server is enough to separate them.

SQLite

SQLite has no change log an external reader can tail: the update hook only fires for writes made through the same in-process connection, and Syncle opens a file that other processes write to. CDC is therefore not supported — the readiness check reports it as such — and the right tool is a watch bridge, which polls and works reliably on SQLite.

Sources behind an SSH tunnel

A connection that uses an SSH tunnel can be the source of a CDC bridge on every engine. The change stream gets a tunnel of its own, opened when the bridge starts and closed when it stops; if the SSH connection drops — a bastion restart, an idle timeout — the stream is stopped, a new tunnel is opened and the stream resumes from its last checkpoint, retrying with a backoff for as long as the bridge is meant to be running. Nothing is lost across the gap: what had not been checkpointed had not been acknowledged to the source either. The bastion's host key is pinned the first time, exactly as for any other use of the connection.

The readiness check

The bridge builder runs a readiness check when you choose the CDC trigger and lists anything missing, with the instruction to fix it. The same probe is available as POST /api/bridges/cdc/readiness with a body of { connectionId, database?, schema?, table }. The response says whether the engine supports CDC at all, whether this connection is ready right now, the individual checks, and the manual steps left:

// request
{ "connectionId": "b6f4…", "database": "shop", "table": "orders" }

// response
{
  "data": {
    "engine": "postgres",
    "supported": true,
    "ready": false,
    "checks": [
      { "label": "wal_level = logical", "ok": false, "detail": "currently \"replica\"" },
      { "label": "role can replicate", "ok": true }
    ],
    "instructions": [
      "Set wal_level=logical on the server (postgresql.conf or your provider’s parameter group) and restart it. This is the one step we can’t automate — it needs a server restart."
    ]
  }
}

Operations and the op token

A CDC trigger carries the set of operations to deliver — any subset of insert, update and delete, all three by default. On a database destination a delete routes to a keyed delete on the target; on an HTTP destination the payload template can expose the operation through the {{$op}} token, which resolves to insert, update or delete. The token is populated on CDC deliveries only — a watch bridge sees rows, not operations; the other template tokens are covered in How bridges work.

PostgreSQL sources have a fourth, opt-in operation: truncate. With it, a TRUNCATE of the source table empties every database target's table (a Redis target has no table to empty, and says so in the delivery's result), and an HTTP destination receives one delivery with an empty row and {{$op}} set to truncate. It is always delivered on its own, in order: rows written before the truncate are delivered before it, and rows inserted after it are still there afterwards. Without it, the truncate is recorded on the timeline as a skipped entry explaining that the destination was left alone. Other engines do not report a truncate as a change (in MySQL it is DDL, not row events), so a bridge on them that asks for it is refused at start instead of waiting for something that never comes.

Going live, stopping, and deleting

A CDC bridge does not run one-shot jobs — asking it to replay returns an error pointing you at live listening instead. Going live opens one long-running job, and every captured change is recorded in it as a delivery, which is what the live timeline shows. Watch bridges share exactly the same lifecycle and the same endpoints:

Canceling the running job stops the listener too — the job pauses and keeps its cursor, so it can resume in place. Updating a live bridge stops the listener first and restarts it on the new configuration. When you go live again, each engine resumes from its persisted cursor: Postgres from the slot's confirmed position, MySQL from the binlog position, MongoDB from the resume token — and Redis always starts from now. Authentication and the response envelope are covered on the HTTP API page.

Deleting a CDC bridge deprovisions what was created for it. On Postgres the replication slot and publication are dropped — this matters, because a slot nothing reads pins WAL on the source and eventually fills its disk. The same happens when a CDC bridge is edited into another kind of bridge or moved to another connection, and a drop that fails is retried until it succeeds (see replication slots and the source's disk). On Redis, notifications are left enabled, since other consumers may rely on them; MySQL has nothing to remove, and MongoDB pre-images stay enabled. Deleting a workspace does the same teardown for every bridge in it.