Syncle

How bridges work

A bridge is the saved sync path: a source, a column mapping, one or more destinations, and a trigger that decides when rows move. This page is the mental model — how a bridge turns into jobs and deliveries, what each trigger mode does, and what Syncle guarantees about what lands on the other side.

Bridges, jobs and deliveries

Three words carry everything here. A bridge is the configuration: the source (a table, optionally filtered and sorted, or a raw query on a connection), the transform, the destinations, and the trigger. A job is one execution of a bridge. A delivery is one row — or one batch, when batching is on — delivered within a job; it is the unit the live timeline shows and the unit you can retry or skip.

Every trigger funnels rows through the same delivery pipeline, so the timeline, retry controls and idempotency behave the same whether a row came from a one-shot replay, a poll, or a change event. Two knobs are narrower than that: batching applies to replay jobs only — watch and CDC always deliver one row per delivery — and the minDelayMs rate limit paces replay and watch deliveries but not CDC.

The three trigger modes

Replay

A replay bridge runs on demand: press Run job and it streams the source once, a page at a time, delivering every row (or the filtered subset). This is the mode for an initial backfill or a one-off migration, and it is the default for a new bridge.

Watch

A watch bridge polls the source on a cursor and delivers whatever is new. Polling works on every engine — including SQLite, which has no change log — so watch is the universal live mode. Three strategies decide what "new" means:

StrategyCursorDetectsSemantics
incrementa strictly-increasing column (auto-increment id, sequence)inserts onlyexact: each poll asks for col > cursor
timestampa created_at / updated_at columnnew rows, plus updates when the column is bumpedpolls col >= cursor with boundary-key dedupe, and re-scans a lookbackMs window (default 3000 ms) behind the cursor so late-committing transactions are not lost
snapshotthe set of primary keys already seenrows with unseen keysfor UUID and other non-monotonic keys; bounded by maxTracked (default 50,000), so best for small and medium tables

The watch trigger itself has three knobs: pollIntervalMs (1000–3600000, default 5000), startFrom (beginning or now, default now — which ignores existing rows and only delivers ones added after the watch starts), and maxPerPoll (default 500), which caps rows delivered per poll cycle as backpressure.

CDC

A CDC bridge streams changes from the database's own change log — real time, no polling: Postgres logical replication, MySQL binlog, MongoDB change streams, Redis keyspace notifications. You can subscribe to a subset of operations (insert, update, delete; default all three). SQLite has no change log, so CDC is not available there — use a watch bridge. Each engine has prerequisites and honest limitations, and the bridge builder runs a readiness check that lists anything missing; the CDC setup page covers all of it.

Delivery guarantees

Database writes are idempotent upserts keyed by columns you choose, using each engine's atomic upsert: ON CONFLICT on Postgres and SQLite, ON DUPLICATE KEY on MySQL, updateOne with upsert on MongoDB. Delivery is at-least-once end to end, and the keyed upsert is what turns that into an exactly-once result — a replayed, retried or redelivered row overwrites itself instead of duplicating. On a CDC bridge, inserts, updates and deletes all propagate; a delete routes to a keyed delete on the target. Watch bridges only surface what polling can see — the table above says which strategy detects what, and none of them detects deletes.

On relational targets a batch is written inside a transaction, so a retried batch is all-or-nothing. MongoDB and Redis have no transaction here; their retry safety comes from the per-row idempotent upsert and delete. Jobs checkpoint progress as they go, survive restarts, and auto-resume after a crash.

Database destinations

A database destination is a list of targets — one bridge can write to several at once. Each target is described by:

FieldDefaultPurpose
connectionIdrequiredthe destination connection
database, schemaoptionalwhere the table lives, when the engine has these levels
tablerequiredtarget table or collection
writeModeupsertupsert writes idempotently keyed by keyColumns; insert always appends
keyColumnsemptytarget columns that uniquely identify a row — required for upsert
mappingempty = identitysource→target column pairs; leave empty to map same-named columns
createMissingTabletruecreate the target table from the source's shape when it does not exist

Auto-created tables use cross-engine type translation: each source column type is collapsed to a portable type (integer, bigint, number, boolean, timestamp, json, uuid, text — falling back to text) and rendered in the target engine's dialect, with keyColumns as the NOT NULL primary key and nothing auto-increment. The mapping preview in the builder renders exactly the mapping the runner performs.

HTTP destinations

Instead of a database, a bridge can POST, PUT or PATCH each row (or batch) to a URL, with optional headers and auth — none, a bearer token, or a custom header, with secrets encrypted at rest. An optional idempotency toggle adds an Idempotency-Key header derived from the job id plus a stable per-delivery identity — the delivery sequence on a replay, the row's key on a watch bridge, the change cursor on CDC — so a redelivery always carries the same key and the receiver can dedupe it.

The request body comes from a JSON template with tokens. The default template is "{{$row}}" — the whole projected row:

Payload template
{
  "event": "row.changed",
  "op": "{{$op}}",
  "row": "{{$row}}",
  "sent_at": "{{$now}}"
}
TokenResolves to
{{column}}the value of that source column
{{$row}}the projected row object, after the optional fields whitelist and rename map
{{$table}}the source table name
{{$op}}the change operation — insert, update or delete — set on CDC deliveries
{{$now}}an ISO timestamp, captured once per delivery
{{$index}}the 0-based row index across the whole job

Substitution happens on the parsed JSON tree, never by string splicing, so a value containing quotes or newlines cannot break the JSON and nothing in a row is ever executed. A string that is exactly one token keeps the value's real type — "{{$row}}" becomes the object itself, not a string — while a token mixed into other text is stringified. Unresolved tokens surface as warnings, never as failures. Outbound requests never follow redirects; the rest of the destination security posture is on the self-hosting page.

Delivery tuning

Every bridge carries the same set of delivery knobs. Three of them — maxAttempts, backoffMs and timeoutMs — govern HTTP deliveries only: a database write is a single attempt, and its retry safety comes from the keyed upsert plus the job-level retry controls. The rest apply to both destination kinds.

KnobRangeDefaultPurpose
batchSize1–10001rows per delivery on replay jobs; watch and CDC always deliver one row
maxAttempts1–103total attempts per HTTP delivery; 1 means no retry
backoffMs0–60000500base retry backoff, doubling each retry up to backoffMaxMs (default 30000)
minDelayMs0–6000000minimum delay between deliveries (replay and watch)
timeoutMs100–12000015000per-request timeout on HTTP deliveries
pageSize1–1000200rows fetched per page from a table source
onErrorcontinue | abortcontinuewhether a failed delivery is logged and skipped past, or aborts the whole job

Job lifecycle and control

A job moves through these statuses:

StatusMeaning
draftprepared in the UI, not sending yet
queuedwaiting in the job queue
runningstreaming and delivering rows
completedran to the end of the source
failedstopped on an error — on a replay job, onError: abort lands here on the first failed delivery; a live watch or CDC bridge pauses instead, keeping its cursor
canceling, canceledcancel requested, then done
pausedstopped by you — resumable in place, as the same job
interrupteda legacy status you may see on old jobs, resumable by hand. A job cut off by a crash keeps its queued or running status and is re-enqueued from its checkpoint at the next boot

Control is in-place: you can cancel a job, resume a paused or interrupted one, skip queued deliveries by range or selection, or retry only the failed rows — the retry re-queues the same job and re-sends just its failed delivery cells, which flip to success in place. All of these are also plain endpoints, documented on the HTTP API page; reading the delivery timeline is covered in the quickstart.

Fan-out and chaining

Because a database destination is a list of targets, one bridge can fan a source out to several databases at once — each target with its own mapping, write mode and key columns. And because any connection can sit on either end, bridges chain: database A feeds B, and a second bridge watches B and feeds C.

Workspaces

Workspaces are the top-level container: every connection and bridge belongs to one. A default workspace always exists, so the concept stays invisible until you create a second one. Deleting a workspace tears down everything in it — CDC slots dropped, watchers stopped, in-flight jobs canceled — before the delete cascades.