How bridges work
A bridge is the saved sync path: a source, a column mapping, one or more destinations, and a trigger that decides when rows move. This page covers how a bridge turns into jobs and deliveries, what each trigger mode does, and what Syncle guarantees about what lands on the other side.
Bridges, jobs and deliveries
Three terms cover most of it. A bridge is the configuration: the source (a table, optionally filtered and sorted, or a raw query on a connection), the transform, the destinations, and the trigger. A job is one execution of a bridge. A delivery is one row — or one batch, when batching is on — delivered within a job; it is the unit the live timeline shows and the unit you can retry or skip.
Every trigger funnels rows through the same delivery pipeline, so the timeline, retry controls and idempotency behave the same whether a row came from a one-shot replay, a poll, or a change event. Two knobs are narrower than that. The bridge's batchSize shapes replay jobs and the requests a CDC bridge sends to an HTTP destination; a watch bridge always delivers one row per delivery, and a CDC bridge writing to a database groups changes on its own (see SYNCLE_CDC_BATCH_SIZE), which idempotent upserts make invisible. And the minDelayMs rate limit paces replay and watch deliveries but not CDC.
The three trigger modes
Replay
A replay bridge runs on demand: press Run job and it streams the source once, a page at a time, delivering every row (or the filtered subset). This is the mode for an initial backfill or a one-off migration, and it is the default for a new bridge.
Only one page is ever in memory, and the pages are read by keyset wherever the engine can do it: “the rows after this one”, which costs the same however deep the read is and lands on the exact next row after a stop, even when rows were added or removed in the meantime. A primary key of any width is the keyset; a sort of the bridge's own gets the key appended behind it and is read the same way, unless a sorted column can hold NULL (a NULL is neither before nor after anything, so that order is read by OFFSET). A table with no primary key is keyed by a unique index whose columns cannot be NULL, when it has one. What is left — a view, a table with neither — is read by OFFSET in the order of all its columns, and the run says so on its timeline: rows added, removed or changed while such a table is being read can be skipped or delivered twice. Give it a key and the read is exact.
On a schedule
A replay bridge can also run by itself. In the builder, under When it runs, choose On a schedule and give it a cron line and the time zone the line is meant in — through the API, trigger: { kind: "replay", schedule: { cron: "0 2 * * *", timezone: "Europe/Rome", enabled: true } }. At every tick the whole source is replayed. Into a target in upsert mode that brings the destination up to date; into one in insert mode it inserts every row again, which the builder warns about — a scheduled bridge almost always wants upsert and key columns.
- The line is the classic five fields —
minute hour day-of-month month day-of-week— with*, lists (1,15), ranges (mon-fri), steps (*/15,9-17/2) and the names of months and days. No seconds, no@daily, noLor#: once a minute is the most a bridge is scheduled. When both day fields are restricted, either one fires it, as in a crontab. The builder shows the next runs as the server works them out, so what you see is what will fire. - The zone is a name (
Europe/Rome,UTC), never an offset: “02:00” stays 02:00 there, summer and winter. On the night the clocks go back the run happens once, not twice; on the night they go forward a run due in the missing hour happens an hour later, not never. - Never two at once. If the run before is still going when the next is due, that tick is skipped — shown on the bridge, and sent as a warning to alert channels subscribed to bridge failures. If it keeps happening, the schedule is tighter than the replay takes.
- From the top, every time. Pressing Run on a bridge whose last run stopped half-way picks that run up again. A scheduled run does not: it is a new run of the whole source, and the stopped one stays in the list as history. Runs the schedule started carry a small calendar mark.
- Off means off. A schedule fires only while it is switched on and the bridge is enabled. A bridge that arrives by import, or as a duplicate, keeps its line but has it switched off, so that a file dropped onto production does not start writing at two in the morning because staging did.
- Restarts. The schedule lives in Redis beside the job queue, and several API processes on one Redis fire it once between them. If the API was down at the time of a tick, that one run happens when it is back; earlier missed ticks are not made up. At every start the schedules in Redis are compared with the bridges in the database and put right.
Watch
A watch bridge polls the source on a cursor and delivers whatever is new. Polling works on every engine — including SQLite, which has no change log — so watch is the universal live mode. Three strategies decide what "new" means:
| Strategy | Cursor | Detects | Semantics |
|---|---|---|---|
increment | a strictly-increasing column (auto-increment id, sequence) | inserts only | exact: each poll asks for col > cursor |
timestamp | a created_at / updated_at column | new rows, plus updates when the column is bumped | polls col >= cursor and re-scans a lookbackMs window (default 3000 ms) behind the cursor so late-committing transactions are not lost. What it has already sent is remembered by key and timestamp, so re-reading the window sends nothing twice — and the same row with a later timestamp is a change, however soon after the last one |
snapshot | the set of primary keys already seen | rows with unseen keys | for UUID and other non-monotonic keys; bounded by maxTracked (default 50,000), so best for small and medium tables. On a Redis source the keyspace is walked by SCAN's own cursor, every key once per poll |
The watch trigger itself has three knobs: pollIntervalMs (1000–3600000, default 5000), startFrom (beginning or now, default now — which ignores existing rows and only delivers ones added after the watch starts), and maxPerPoll (default 500), which caps rows delivered per poll cycle as backpressure.
CDC
A CDC bridge streams changes from the database's own change log — real time, no polling: Postgres logical replication, MySQL binlog, MongoDB change streams, Redis keyspace notifications. You can subscribe to a subset of operations (insert, update, delete; default all three — PostgreSQL sources can also mirror TRUNCATE, which is opt-in). SQLite has no change log, so CDC is not available there — use a watch bridge. Each engine has prerequisites and documented limitations, and the bridge builder runs a readiness check that lists anything missing; the CDC setup page covers all of it.
Copy what is there, then follow
A change log only knows about changes. A CDC bridge set to Only changes from now on (the default) leaves the rows a table already holds where they are. Set to Copy what is there, then follow changes, the same bridge does both — and the point of doing both in one bridge is the seam between them. A replay followed by a separate CDC bridge either misses what changed in between, or lets an old value the replay read land on top of a newer one. So, on its first start, the bridge:
- takes its place in the source's change log — without reading from it yet;
- reads the table from one end to the other and delivers every row as an
insert, through the same pipeline changes use (filters, transforms, batching, the dead-letter queue); - opens the change stream at the place it took in step 1. Everything that changed while the copy ran arrives now, after the copied rows, in order.
A row that changed during the copy is therefore delivered twice — once as the copy read it, once as the change — and ends up right, because a keyed target is written with an idempotent upsert. An append-only (insert-mode) target or an HTTP receiver sees it twice; that is the same at-least-once contract every live bridge has. The timeline marks the end of the copy with a note ("Copied the 1,204,551 rows the table already had…").
- It resumes. The copy's progress is checkpointed like any other position: a bridge stopped — or a Syncle restarted — mid-copy carries on from the row it had reached, with the place in the log it took the first time.
- It happens once. A bridge that already has a position resumes from it; changing the setting later does not re-copy anything. The table needs a primary key, or a unique index whose columns cannot be
NULL, to be copied in a stable order while changes arrive under it — without one the start is refused, before anything is touched (a replay reads such a table byOFFSETand says so; a copy that a log is replayed on top of has to be exact). - The source holds the log for as long as the copy runs. On PostgreSQL the replication slot pins WAL from the moment it is created (the slot guard reports how much); on MySQL and MongoDB the binlog / oplog has to outlast the copy, or the bridge stops and says its position was lost. Redis has no log: the stream is opened first and its changes are held in memory — the newest per key — until the copy is done (see
SYNCLE_SNAPSHOT_HOLD_MAX). - After a lost position, such a bridge offers to copy the table again, which brings every row that still exists up to date. Rows deleted at the source in the meantime stay at the destination. "Continue from now" still means exactly that: no copy.
Filters and column transforms
Two things can happen to a row between the source and the destination: it can be left out, and its values can be changed. Both are set in the bridge builder, both are part of the bridge's saved configuration, and both apply the same way whatever the trigger and whatever the destination.
Only rows where…
A bridge's source can carry a list of conditions — equals, is not, greater / less than, contains, starts / ends with, is empty, is not empty — and a row is sent only when it meets all of them. A replay and a watch bridge push the conditions into the source's own query, so rows that do not match are never read; a CDC bridge sees every change of the table and applies the same conditions, with the same semantics, to each one. A value typed into the builder for a numeric or boolean column is sent as a number or a boolean, which is what MongoDB needs to match it.
The builder does not save a condition that is only half written — a comparison with no value blocks the save instead of quietly becoming "no condition", which would send every row. Conditions set through the API that the editor has no row for (an in list, for instance) are kept as they are when the bridge is edited.
Change values on the way
A bridge can carry a list of steps that are applied to every row before it is delivered. There are five kinds:
| Step | What it does |
|---|---|
| Mask | hides a value. Keep the ends shows the first / last few characters and fills the middle (a value too short to keep anything of is filled entirely); redact replaces it with eight fill characters, whatever its length; hash replaces it with its SHA-256 (hex), optionally salted — stable, so the column still joins, dedupes and works as a key; empty sends NULL and keeps the column |
| Convert type | to text, number, whole number, true / false, date and time, or JSON. "yes", "1" and "on" are true; a number of seconds or milliseconds since 1970 is a date |
| Clean text | trim, lower case, upper case |
| Default value | a value for when the source has none (typed in the builder for a numeric or boolean column, it is sent as a number or a boolean) |
| Computed column | a column built from a template over the others: {{first_name}} {{last_name}}. {{$now}} and {{$table}} are available. A template that is exactly one token copies that column with its type. A name the table does not have adds a column |
Steps run top to bottom and each one sees what the ones above it did, so "lower-case the e-mail, then hash it" is two steps in that order. They are declarative on purpose: there is no expression language and nothing is evaluated — a bridge's configuration runs inside the process that holds every stored credential.
A value that cannot be converted ("n/a" to a number) is never decided quietly. Each conversion says what happens: fail the delivery (the default), send it as empty, or leave it as it was. A failed conversion fails the delivery before anything is sent, with the column and the value in the error — on a bridge set to continue, that one row goes to the dead-letter queue and the rest carry on. A retry from there re-reads the row and runs the steps again, so fixing the value at the source is enough.
What follows from changing values on the way:
- A table Syncle creates fits the new values. A hashed integer is text; a column converted to a date is a timestamp; a plain copy (
{{price}}) is typed like the column it copies; a column a step can empty is nullable even when the source saysNOT NULL; and the columns the steps add are created with the rest. The dry run shows the planned column types, and the shaped sample rows, before anything exists. An existing table is never altered — its columns have to be able to hold what the steps produce. - A masked key still works. A delete arrives with the source's plain key; the same steps are applied to it, so it finds the row that was written under the hash. Computed columns and defaults are not invented for a row that is being deleted.
- Delivery records hold the shaped row. A masked value does not reappear in the job timeline or in the payload Syncle keeps of an HTTP delivery.
- Masking protects the destination, not Syncle's own queue. A row set aside in the dead-letter queue is kept as the source has it — that is what lets a retry re-read and re-shape it. Anyone who can open Syncle can already browse the source.
- On PostgreSQL CDC, a large column that an update did not touch is not resent by the server. When a step needs such a column (a computed column that reads it), Syncle reads the row back from the source rather than compute from nothing.
Delivery guarantees
Database writes are idempotent upserts keyed by columns you choose, using each engine's atomic upsert: ON CONFLICT on Postgres and SQLite, ON DUPLICATE KEY on MySQL, updateOne with upsert on MongoDB. Delivery is at-least-once end to end, and the keyed upsert is what turns that into an exactly-once result — a replayed, retried or redelivered row overwrites itself instead of duplicating. On a CDC bridge, inserts, updates and deletes all propagate; a delete routes to a keyed delete on the target. Watch bridges only surface what polling can see — the table above says which strategy detects what, and none of them detects deletes.
On relational targets a batch is written inside a transaction, so a retried batch is all-or-nothing. MongoDB and Redis have no transaction here; their retry safety comes from the per-row idempotent upsert and delete. Jobs checkpoint progress as they go, survive restarts, and auto-resume after a crash.
Database destinations
A database destination is a list of targets — one bridge can write to several at once. Each target is described by:
| Field | Default | Purpose |
|---|---|---|
connectionId | required | the destination connection |
database, schema | optional | where the table lives, when the engine has these levels |
table | required | target table or collection |
writeMode | upsert | upsert writes idempotently keyed by keyColumns; insert always appends |
keyColumns | empty | target columns that uniquely identify a row — required for upsert |
mapping | empty = identity | source→target column pairs; leave empty to map same-named columns |
createMissingTable | true | create the target table from the source's shape when it does not exist |
onDelete, softDelete | delete | what a delete at the source does to this target — see below |
redis | none | a target in Redis: how a row becomes a key — see Rows in Redis |
Rows in Redis
Redis has no tables, so a target there says how a row becomes a key. Pick a Redis connection in the builder and the target card asks for exactly that:
| Field | Default | Purpose |
|---|---|---|
redis.keyTemplate | required | the key, built from the row's (target) columns: user:{{id}}, tenant:{{tenant_id}}:user:{{id}}. It has to contain at least one column, and only columns — {{$now}} is refused, because the same row has to be the same key every time |
redis.type | hash | hash: a field per column, so HGETALL gives the row back. A NULL column is a field that is not there, and fields your application adds beside the row's are left alone. json: the whole row as one JSON document in a string (NULLs included). string: one column's value |
redis.valueColumn | — | string only: the column whose value is stored |
redis.ttlSeconds | never | the key expires this long after its last write — every update renews it. Without it the key does not expire (an expiry it had is removed) |
The columns in the key are the target's key columns — you do not pick them separately. So a delete at the source removes that key (or leaves it, with onDelete: ignore; a key cannot be marked deleted, so soft is refused), an UPDATE that changes a key column moves the row to its new key and removes the old one, and a row that has no value for a key column fails — by name — instead of landing under half a key. A key that already exists as another type is replaced.
Values are kept as text, as Redis keeps everything: numbers as their digits, booleans as true/false, a JSON column as its JSON, binary as the bytes themselves (in a JSON document: base64), and a moment in time as ISO-8601 in UTC — 2026-03-01T10:20:30.123000Z, to the microsecond the source keeps — which is what date parsers take.
When a row is deleted at the source
A live bridge that captures deletes applies them, and each target says how — so one bridge can keep a working copy in step and feed an archive that never forgets:
- Delete it here too (
delete, the default): a keyed delete. - Keep it, and mark it as deleted (
soft): the row stays, and a column of the target —softDelete.column,deleted_atby default — is set to the time of the delete, or totruewithsoftDelete.value: "boolean". Every ordinary write sets it back toNULL/false, so a row that comes back at the source under the same key is unmarked by the write that brings it back. The marker is a column of its own: not a key, not one that receives source data. Syncle creates it with the table when it creates the table; an existing table has to have it already — the dry run says so when it does not — because an existing table is never altered. - Do nothing (
ignore): the target keeps every row it was ever sent.
A TRUNCATE (PostgreSQL sources, opt-in) empties only the targets that delete; a soft-delete or ignore target is left as it is, and the delivery's summary says so per target. A target with no key columns (append-only insert mode) has nothing to find a row by, so deletes are never applied to it, whatever the policy. Polling (watch) bridges cannot see deletes at all.
How column types are translated
An auto-created table takes keyColumns as its NOT NULL primary key, nothing auto-increment, and every other column typed from the source. Between two instances of the same engine the source's own type is reused word for word — numeric(38,10) stays numeric(38,10), timestamp(3) with time zone stays exactly that — so a same-engine copy narrows nothing. (The exception is a type that lives in one database only, such as a Postgres enum or domain: a domain becomes the base type it wraps, an enum becomes text.)
Across engines each type is read in the source engine's dialect — float is single precision in MySQL and double in Postgres, timestamp means different things in each — and rendered as the closest type the target has:
| Source column | → PostgreSQL | → MySQL | → SQLite |
|---|---|---|---|
exact decimal, numeric(p,s) | NUMERIC(p,s) | DECIMAL(p,s), up to (65,30) | DECIMAL(p,s) to 15 digits, text beyond |
64-bit and unsigned integers (bigint unsigned) | BIGINT / NUMERIC(20,0) | as declared | INTEGER / text |
boolean, MySQL tinyint(1) and bit(1) | BOOLEAN | TINYINT(1) | INTEGER |
bytes (bytea, blob, binary) | BYTEA | LONGBLOB | BLOB |
| unbounded text | TEXT | LONGTEXT (MySQL's TEXT stops at 64 KB) | TEXT |
| wall-clock timestamp (no zone) | TIMESTAMP | DATETIME(6) | text |
instant (timestamptz, a MongoDB date) | TIMESTAMPTZ | DATETIME(6), as UTC | ISO-8601 text, UTC |
| JSON, arrays, nested documents | JSONB; a Postgres array stays an array | JSON | JSON text |
uuid / MongoDB ObjectId | UUID / VARCHAR(24) | CHAR(36) / CHAR(24) | text |
| enum, set, geometry, anything Syncle does not recognise | text, with a warning | ||
MongoDB has no declared types, so a field's type is sampled from its documents — all of the sample, not just the first, so a field that starts out null still gets its real type, and one that holds both numbers and strings is stored as text.
Narrowing is always reported. Whenever the target cannot hold everything the source column can — a time zone MySQL has nowhere to put, a precision beyond MySQL's 65 digits, a key column that had to be bounded to VARCHAR(255), an enum carried as text — the column is named in a warning. The preview (POST /api/bridges/:id/preview) lists the exact columns a run would create and those warnings before anything runs, and the same warnings are logged when the table is created. If a default is not what you want, create the table yourself: an existing table is never altered.
Previewing a run
The builder's Dry run button shows what the bridge would do as it is set up right now, before it is saved: for each database target, whether the table exists and — if a run would create it — the table column by column, with the source's type beside the type it becomes; every column the target cannot hold faithfully, as a warning; and a few real rows from the source as they would be written (or, for an HTTP destination, the rendered payloads with the auth secret redacted). Nothing is saved, no table is created, and nothing is delivered. The same check is available for a saved bridge from the API — see the preview endpoints.
How values keep their precision
Picking the right column type is only part of it: the drivers on either side also have to agree on what a value is. Syncle reads values in whichever form loses nothing, and converts only where the target needs it:
- Postgres dates and timestamps are read as the text Postgres sends, not as JavaScript dates — so microseconds survive (every
now()-stamped column has them), and a wall-clocktimestampcannot shift by the server's time zone. An instant is written to MySQL and SQLite as its UTC reading. - Exact numbers stay text end to end: Postgres
numericandbigint, MySQLDECIMALandBIGINT(from the binlog too), SQLite 64-bit integers, MongoDBDecimal128andLong. - MongoDB's wrapper types become plain values at every depth of a document: an
ObjectIdits hex string, a binary bytes, a UUID its canonical string. - A MySQL zero date (
0000-00-00) is MySQL's "no date" and becomesNULLin any other engine. Atinyint(1)orbit(1)becomes a real boolean. - JSON keeps its shape into Postgres whatever it holds: an empty array stays an array, and the JSON string
"123"stays a string.
A row reads the same whether it arrived by replay or by CDC — the two are tested against each other on real engines, under more than one server time zone. One thing cannot be told apart: a JSON null inside a Postgres json column reads the same as SQL NULL, and is written as NULL.
HTTP destinations
Instead of a database, a bridge can POST, PUT or PATCH each row (or batch) to a URL, with optional headers and auth — none, a bearer token, or a custom header, with secrets encrypted at rest. An optional idempotency toggle adds an Idempotency-Key header derived from the job id plus a stable per-delivery identity — the delivery sequence on a replay, the row's key on a watch bridge, the change cursor on CDC — so a redelivery always carries the same key and the receiver can dedupe it.
The request body comes from a JSON template with tokens. The default template is "{{$row}}" — the whole projected row:
Payload template
{
"event": "row.changed",
"op": "{{$op}}",
"row": "{{$row}}",
"sent_at": "{{$now}}"
}| Token | Resolves to |
|---|---|
{{column}} | the value of that source column |
{{$row}} | the projected row object, after the optional fields whitelist and rename map |
{{$table}} | the source table name |
{{$op}} | the change operation — insert, update or delete (or truncate, where a PostgreSQL bridge captures it) — set on CDC deliveries |
{{$now}} | an ISO timestamp, captured once per delivery |
{{$index}} | the 0-based row index across the whole job |
Substitution happens on the parsed JSON tree, never by string splicing, so a value containing quotes or newlines cannot break the JSON and nothing in a row is ever executed. A string that is exactly one token keeps the value's real type — "{{$row}}" becomes the object itself, not a string — while a token mixed into other text is stringified. Unresolved tokens surface as warnings, never as failures. Outbound requests never follow redirects; the rest of the destination security posture is on the self-hosting page.
Delivery tuning
Every bridge carries the same set of delivery knobs. Three of them — maxAttempts, backoffMs and timeoutMs — govern HTTP deliveries only: a database write is a single attempt, and its retry safety comes from the keyed upsert plus the job-level retry controls. The rest apply to both destination kinds.
| Knob | Range | Default | Purpose |
|---|---|---|---|
batchSize | 1–1000 | 1 | rows per delivery on replay jobs, and per request for a CDC bridge with an HTTP destination; watch delivers one row at a time |
maxAttempts | 1–10 | 3 | total attempts per HTTP delivery; 1 means no retry |
backoffMs | 0–60000 | 500 | base retry backoff, doubling each retry up to backoffMaxMs (default 30000) |
minDelayMs | 0–600000 | 0 | minimum delay between deliveries (replay and watch) |
timeoutMs | 100–120000 | 15000 | per-request timeout on HTTP deliveries |
pageSize | 1–1000 | 200 | rows fetched per page from a table source |
onError | continue | abort | abort | what a failed delivery does — stop without moving past it, or set the failed rows aside and carry on. See When a delivery fails |
onSchemaChange | stop | evolve | continue | stop | what happens when the source table is no longer the one the bridge was built on. See When the source table changes |
When a delivery fails
One rule sits under everything here: a row Syncle has read is always in one of three places — the destination, the bridge's dead-letter queue, or still ahead of the bridge's cursor, where it will be read again. It is never in none of them. That matters most on a CDC bridge, because a change stream is read once: when Syncle confirms a position, the source is free to discard everything before it, and a row that was stepped over cannot be fetched a second time.
onError picks between the two ways of honouring that rule.
onError | On a failed delivery | Choose it when |
|---|---|---|
abortthe default when a bridge is created through the API without saying | the bridge stops at the failure, cursor untouched. A live bridge pauses and a replay job fails, each with the reason. Start it again and the same rows are retried into the same timeline cell. | the destination must never run ahead of a row it is missing, and someone will notice a stopped bridge |
continue | the bridge keeps going. On a live bridge the rows that failed are first written, complete, to the dead-letter queue — only then does the cursor move. | one bad row should not hold up every row behind it. This is what the web app's builder pre-selects; the choice is on the form either way |
The dead-letter queue
A database batch is a single transaction, so one row the destination refuses — a value a constraint rejects, a type it cannot hold — fails every row that shared its batch. Under continue Syncle does not set the whole batch aside: it splits it and re-delivers the halves, again and again, until each failure is pinned to a single row. The healthy rows land; only the rows actually at fault are queued. (Finding one bad row among a hundred thousand costs a few dozen attempts, not a hundred thousand.) An HTTP destination is not split — there the batch is the payload the receiver sees — so the failed request is queued whole, and the same goes for a target in insert mode, where re-delivering rows would append them twice.
An entry holds the source row exactly as it was read: never truncated, with bytes, timestamps and 64-bit integers preserved. Retrying it is safe while the bridge is still streaming, because a retry does not replay the recording. For a database destination fed from a table with a primary key, Syncle looks the row up at the source again and writes what is there now: the current row if it exists, a delete if it is gone, nothing if it has since left the bridge's filters. Replaying an hours-old payload could overwrite a newer version the stream has delivered in the meantime; re-reading cannot. The recording itself is sent only where that is the right thing — to HTTP destinations, to append-only insert targets, and for sources with no primary key to look a row up by.
One case cannot be settled automatically: the source row is gone, and the bridge does not propagate deletes (a watch bridge, or a CDC bridge with deletes switched off). The recording might be the newest version of that row or an outdated one, and nothing can tell which, so the entry waits. Retry it with force to write the recording anyway, or discard it.
When every row from a failed delivery has been delivered, its timeline cell turns green and the job's counters follow. Discarded rows leave the cell red — they never arrived.
When continue stops anyway
continue is for bad rows, not for a broken destination. If the target is unreachable or its table is gone, every row fails, and carrying on would simply pour the change stream into the queue. So a continue bridge still stops — cursor untouched, nothing lost, the reason on the job — when:
- splitting a failed batch shows the failure is not confined to a few rows (nothing at all succeeds, or more than 100 rows of one batch are bad);
- several batches in a row deliver nothing (
SYNCLE_MAX_CONSECUTIVE_FAILURES, default 5); - the queue is full (
SYNCLE_DEAD_LETTER_MAX_ROWS, default 10,000 waiting rows per bridge); - the failed rows could not be written to the queue.
Fix the destination, start the bridge again, and retry whatever was queued before it stopped.
Verify and reconcile
A bridge that has streamed for a month has delivered millions of changes, each of them green. That is evidence the pipe works — not that the two ends agree. A row edited by hand at the destination, a delete that happened while the bridge was paused past the source's log, a target restored from last week's backup: none of those is a failed delivery. Verify (the magnifier on a bridge's page, or POST /api/bridges/:id/verify) looks:
- the source is read once, a page at a time. Each page is turned into the rows the bridge would write — same filters, column transforms, value conversion and mapping, by the code that writes them — and the destination is asked for the rows with those keys. That finds what is missing and what is different;
- the destination is read once, and the source asked for its keys. That finds what is only in the destination.
Values are compared by what kind of value the column holds, not as text: a numeric that one driver hands over as '1.50' and another as 1.5 is the same number, a timestamptz is the same instant in any zone's spelling, jsonb is the same document whatever its key order. A TEXT key is never read as a number (007 and 7 are two rows). What is not forgiven is a destination column narrower than the source — fewer decimals, no fractional seconds: those rows are different, and saying so is the point. A computed column that uses {{$now}} can never be the same twice and is left out, with a note.
A bridge that is still delivering is a moving target. A row read a moment before its change arrives looks different without being wrong, so nothing is reported on a first reading. What looks wrong is read again from both ends a little later (SYNCLE_VERIFY_RECHECK_MS, default 1.5 s), and only what is still wrong is reported.
Reconcile does the same and writes the rows that are missing or different — from the source as it is at that second look, through the bridge's own sink, and checks once more that the repair was not overtaken by the stream. Rows that are only in the destination are removed only when you tick the box (deleteExtra), and then the way the target's delete policy says: deleted, or marked. For a target that ignores deletes they are not even looked for — they are what such a target is for.
When the source table changes
A bridge is built against the columns a table has on the day it is built, and the table goes on living. Syncle keeps the columns the bridge was built for — recorded when the bridge is created, and again every time it is saved — and compares them with the table before a run starts, when a live bridge starts, and — on a live bridge and during a replay alike — the moment a row arrives whose columns are not the ones the row before it had. A dead-letter retry asks first as well: it re-reads rows from the source, and is refused while the answer is “a column this bridge uses is gone”.
Two very different things can have happened:
- A column the bridge uses is gone — dropped, or renamed, which to a catalog is the same thing. “Uses” means: mapped by name to a destination column, a key of a target with no mapping, filtered or sorted by, read by a column transform, polled by, pinned in an HTTP payload's field list, or named in its template (
{{email}}). Every row from then on has no value for it, and an upsert would writeNULLover the value the destination holds, row by row, every delivery green. So the bridge stops before that write: the job fails with the column's name, the row is not delivered and not skipped (a live bridge reads it again when it next starts), and the destination is exactly as it was. - Anything else — a column added, a type changed, a column the bridge never touched dropped. The bridge carries on. Its page shows what changed, with an Accept button, because a copy that is now narrower than its original is worth knowing about.
The way out of a stop is to edit the bridge: re-map the column to its new name, or remove it. Saving a bridge that no longer uses a missing column accepts the table as it is now. Saving it unchanged does not, and neither does Accept — there is deliberately no button that turns the protection off for one column.
onSchemaChange | A used column is gone | A column was added |
|---|---|---|
stop (default) | stops before writing; critical alert | noted on the bridge; warning alert; not copied |
evolve | stops before writing; critical alert | added to every destination table that Syncle creates and fills without a column mapping — typed by the same map an auto-created table uses, always nullable. A target whose columns you chose is left alone. Nothing is ever dropped or retyped at the destination |
continue | carries on, writing NULL for the column — the behaviour before this setting existed; warning alert | noted on the bridge; warning alert; not copied |
Each change is said once — in the log, and to every alert channel subscribed to a source table changes under a bridge — not once per batch.
Job lifecycle and control
A job moves through these statuses:
| Status | Meaning |
|---|---|
draft | prepared in the UI, not sending yet |
queued | waiting in the job queue |
running | streaming and delivering rows |
completed | ran to the end of the source |
failed | stopped on an error — on a replay job, onError: abort lands here on the first failed delivery; a live watch or CDC bridge pauses instead, keeping its cursor |
canceling, canceled | cancel requested, then done |
paused | stopped — by you, or by a failure on a live bridge (the reason is on the job). Resumable in place, as the same job, from the same cursor |
interrupted | a legacy status you may see on old jobs, resumable by hand. A job cut off by a crash keeps its queued or running status and is re-enqueued from its checkpoint at the next boot |
Control is in-place: you can cancel a job, resume a paused or interrupted one, skip queued deliveries by range or selection, or retry only the failed rows — the retry re-queues the same job and re-sends just its failed delivery cells, which flip to success in place. On a live bridge the failed rows are in its dead-letter queue, and Retry failed retries that instead, without stopping the bridge. A replay that had stopped at a failure (on failure: abort, the default) does not end there: once the failed delivery has gone through, it carries on from where it stopped, with the bridge as it is configured now — the rows after the failure were never read, and completed has to mean them too. A single failed delivery can be retried on its own from its detail panel, a job's failures can be downloaded as CSV or NDJSON (the rows' keys, the error, the payload that was sent), and a bridge with more than one run shows them as a strip above the timeline — green for a clean run, amber for one that finished with failed deliveries, red for one a failure stopped. All of these are also plain endpoints, documented on the HTTP API page; reading the delivery timeline is covered in the quickstart.
Duplicate, export, import
Duplicate makes a copy of a bridge under a new name, in the same workspace — credential and all, since it never leaves the instance. The copy has no job and no position: a copy of a live bridge starts from scratch when it is started.
Export downloads a bridge (or, from the API, a whole workspace) as a JSON document: to keep in version control, to move from staging to production, to hand to a colleague. What travels is configuration — source, filters, transforms, targets, delete policy, delivery, trigger. What never does is anything secret:
- an HTTP destination's token or header value leaves empty. A bridge imported without its credential arrives switched off, and says why;
- a connection is a reference — its id, with its name and engine beside it. Where it points, and as whom, is not part of a bridge and is not exported.
Import has to find this instance's connections for the ones the file talks about: the same id (a re-import where it came from), else the only connection here with that name and engine. When neither settles it, Syncle asks — listing your connections of that engine — and creates nothing until every one is decided: half an import is worse than none. Names that are taken get (imported) appended.
Fan-out and chaining
Because a database destination is a list of targets, one bridge can fan a source out to several databases at once — each target with its own mapping, write mode and key columns. And because any connection can sit on either end, bridges chain: database A feeds B, and a second bridge watches B and feeds C.
Two-way sync, and rings
Two live bridges can feed each other — A → B plus B → A — and keep two tables in step in both directions. Left to itself that arrangement never rests: A's change is written to B, which B's change log reports, which the other bridge writes to A, which A's log reports… On PostgreSQL an upsert of identical values is still a logged change, so one INSERT by a person went back and forth about nine times a second for as long as both bridges ran.
Syncle stops it by knowing its own writes. When a bridge writes to a table that another listening bridge reads, what it is about to write is remembered — per table, per row, in order — in Syncle's own Redis. The bridge that reads that table looks every change up there; one that matches is this instance's own write coming back, and:
- A pair (A ⇄ B): the change crosses once and is not sent back. Inserts, updates, deletes and mirrored
TRUNCATEs alike. - A chain (A → B → C) keeps working: a recognised change remembers which tables it has been through, and is only kept from going back to one of them. B's bridge to C passes on exactly the rows A's bridge wrote.
- A ring (A → B → C → A) stops where it began, whichever table the change was made in.
It needs nothing from the source — no replication origins, no marker columns, no triggers — and works across engines and for polling (watch) bridges as well as CDC. Values are compared as what they are (12.50, '12.5' and 12.5 are one number), the same way Verify compares them. A bridge that is tied to another says so on its page, names it, and counts the changes it held back.
Two more things follow from it. Before writing to such a table the bridge looks at what is there, and a row that is already exactly what it would be set to is not written (the delivery says wrote 0 (3 already up to date, not written)). That is also the safety net: a change that is not recognised — the memory of it expired because the reading bridge was further behind than SYNCLE_ECHO_TTL_SECONDS (default 5 minutes) — is sent on, finds the other side already up to date, writes nothing, and the loop dies by itself one hop later. That look does not go through Redis, so it holds while Redis is away too. And none of this costs anything for a bridge whose destination nobody reads: no look before the write, nothing kept in Redis.
What is remembered is a row of yours, so it is kept the way your connection passwords are: encrypted under the master key, used up the moment the change comes back, and gone after SYNCLE_ECHO_TTL_SECONDS if it never does. Large values (a document, a file in a column) are remembered by their SHA-256 only.
What it does not do:
- Resolve conflicts. If the same row is edited on both sides within the same moment, each edit crosses to the other side and the two tables can end up holding each other's value. Nothing loops, and nothing is flagged as failed — run Verify on a schedule if that matters, and let one side own each row where you can.
- See through columns the database changes on every write. A trigger that stamps
updated_at = now()on both sides makes every copy differ from what was written, so it is never recognised and each hop changes the row again. Leave such columns out of the mapping (both bridges), or let only the application set them. - Know about writers that are not Syncle. A loop through a webhook and somebody else's code (A → HTTP → their service → B → A) is theirs to break.
Workspaces
Workspaces are the top-level container: every connection and bridge belongs to one. A default workspace always exists, so the concept stays invisible until you create a second one. Deleting a workspace tears down everything in it — CDC slots dropped, watchers stopped, in-flight jobs canceled — before the delete cascades.
