Why Transaction Logs Matter: The Idea That Became Remac
Editor’s note, July 2026: I first published this article while the prototype used the working name Stream. That work became Remac. I have revised the article to reflect the system that grew from it while preserving the original question that started the work.
I have worked on systems where a database committed an important change, but the next system did not see it until much later. A message relay had stalled. A polling job had not started. An outbox worker needed another retry. A repair process had to compare two systems and decide which one was correct.
The database had already accepted the transaction. The delay existed between that commit and the systems that needed to act on it.
Teams solve this problem in many ways. I have used transactional outboxes, database queues, PostgreSQL LISTEN/NOTIFY, trigger and event tables, reconciliation jobs, etc. These approaches can work. They also add code, state, and failure paths that somebody must own.
The recurring problem led me to a simpler question:
What useful work can we build on the transaction log that the database already creates?
That question led to an early prototype in late 2025. A few weeks later, the prototype could move committed changes from PostgreSQL to a JSONL file. It had no checkpoints, no optimized architecture, and no external processing stack. It proved one narrow point: basic real-time data movement did not require a large collection of supporting services.
The harder work started after that result.
A transaction log is more than a list of changed rows
A database transaction log exists first for the database itself. It helps the engine preserve durability, recover after a failure, and support replication or backup functions.
The format and behavior depend on the database. PostgreSQL writes a Write-Ahead Log. MySQL uses a binary log for replication and recovery functions. MongoDB replica sets maintain an operations log. Oracle uses redo logs.
These logs are not interchangeable. They do not expose the same information, position model, retention behavior, or recovery rules.
They are also not row-event feeds by default.
PostgreSQL WAL records changes at the storage level. Logical decoding converts selected persistent table changes into a form that another system can understand. For PostgreSQL, logical replication requires wal_level to be set to logical. It also requires the correct privileges, replication resources, publication rules, and replica identity for the selected operations. PostgreSQL documents these requirements directly.
MongoDB change streams require a replica set or sharded cluster. Their ability to resume also depends on the required history remaining in the oplog. MongoDB documents the deployment and retention boundaries.
A transaction log processor must respect these differences. It cannot assume that every database exposes every processing mode after installation.
The log also has an information boundary. It records what the database engine records. It does not automatically contain the business reason for a change. It does not record every read. Some old row values, schema operations, sequences, and database-specific objects can require separate handling or might not appear in a logical change stream.
The useful foundation is still strong. Within a declared scope, the log can provide committed changes, an ordered position model, transaction context, and recovery history. A processor must convert that foundation into explicit behavior.
Where batch and polling begin to fail
Batch processing remains a valid choice when a system can tolerate its update interval. A nightly warehouse load can be correct for a report that is read once each morning.
The fit changes when freshness, deletion handling, transaction context, or recovery matters.
Consider a polling job that asks each table for rows changed after a saved timestamp. That design must trust the timestamp, the clock, the query boundary, and every write path that updates the field. A missing timestamp update can hide a change. A deleted row no longer exists for the next query to find. Two tables changed by one transaction can reach the destination through separate jobs.
More careful polling designs can solve some of these problems. A high-water mark can define progress. Tombstone tables can expose deletes. Triggers can record changes. Reconciliation can detect drift.
Each addition creates another contract:
- Which write path creates the tombstone?
- When can the high-water mark advance?
- What happens when the destination accepts half of the work?
- How does the job resume after a crash?
- Which system wins when reconciliation finds a difference?
The system can answer all of these questions. The answers are part of the product, whether the team writes them down or not.
Transaction-log processing starts from the database’s committed history instead of reconstructing that history from the current contents of a table.
That change in starting point matters.
The outbox solves a different part of the problem
The transactional outbox is useful because it can write application data and an event record in one database transaction. If the transaction commits, both records commit. A relay can then deliver the event to a broker or another system.
I used this pattern because it solved real dual-write problems. It also required each service to own an outbox schema, integration code, relay behavior, retries, cleanup, and monitoring. At scale, outbox tables can create growth, query contention, and database maintenance work.
The outbox still has a property that a transaction log cannot invent: the application can record domain intent.
An event named OrderApproved tells a consumer why the application changed its data. A row update in an orders table tells the consumer what the database committed. The two records can describe the same business operation, but they are not the same fact.
A transaction log processor does not replace every outbox. It can remove the need to make every service build its own data-change relay. It can also carry an outbox event after the application records one. The correct choice depends on whether the consumer needs committed row changes, domain events, or both.
CDC is one use of the log
Change Data Capture asks a focused question:
Which selected row changes committed after a known source position?
A log-based CDC system decodes those changes and delivers them to configured destinations in real time. It can avoid repeated table scans and capture deletes that timestamp polling can miss.
Reading the log does not provide correctness by itself. The processor still has to handle transaction assembly, ordering, retries, checkpoints, acknowledgments, backpressure, large transactions, source retention, sink failures, and safe recovery.
This is where my view of the product changed.
The first prototype looked like a small CDC path because it moved PostgreSQL changes to a file. The underlying resource supported a much larger system. CDC was one function of transaction log processing, not the boundary around it.
Remac is built around that distinction.
Two processing modes, six functions
Transaction logs support two different processing modes.
Logical processing converts selected committed changes into normalized events. These events can carry available source, table, operation, row, position, time, and transaction context. The processor can filter, transform, route, archive, and apply them through declared policies.
Physical processing keeps the source database’s native log bytes or files. Those records remain tied to the source engine and its recovery rules. They support compatible full-cluster backup and recovery operations.
Remac uses these two modes to support six related functions.
1. Change Data Capture
Remac converts selected committed changes into row-level events and delivers them to configured downstream sinks in real time.
2. Logical replication
Remac applies selected decoded changes to another database that uses the same engine. The declared row state at the destination converges with the selected source state.
Sending an event to Kafka, RabbitMQ, or a file is event delivery. Applying PostgreSQL changes to PostgreSQL row state is logical replication.
3. Heterogeneous replication
Remac maps and applies selected committed changes to a database that uses a different engine. The destination uses its own type system and write behavior, while the selected row state converges under the declared mapping.
An object store or message broker can receive events, but it is not a heterogeneous replication database.
4. Audit logging
Remac preserves selected committed changes and their available technical context in an append-oriented record. This can show how selected data changed.
Database change history does not replace application access logs, identity records, or every control required by a compliance program.
5. Point-in-time recovery
Physical PITR combines a compatible base snapshot with the source database’s native log. Recovery can restore a full cluster to a target covered by the available snapshot and continuous log history.
Logical recovery has a different scope. It can replay an archived event stream to reconstruct selected row state or a declared projection. It does not restore every physical property of a database cluster.
6. REDO and event replay
Remac can replay archived normalized events through configured sinks. This supports rebuild, repair, testing, and reconstruction of declared downstream state.
These functions share a foundation, but they do not share one loose definition. Each function has its own scope, position rules, destination behavior, and recovery boundary.
One transaction log. Six functions. One contract.
Moving events is the easy demonstration
The first prototype proved that PostgreSQL changes could reach a file without a separate processing cluster. It did not prove that the path was complete, ordered, resumable, or recoverable under every relevant failure.
Those properties require deliberate engineering.
A processor can read every selected event and still produce the wrong destination state if it changes commit order. It can deliver an event and still lose it if it advances a checkpoint before the sink’s confirmation is safe. It can report high throughput while an unbounded queue transfers risk into memory. It can resume from a stored position and still create a gap if earlier work remains incomplete.
Remac defines these requirements through the Remac Contract:
- Completeness
- Ordering
- Delivery
- Resumability
- Consistency
- Liveness
- Recoverability
The numbers identify the guarantees. They do not rank them.
The contract forces each processing mode to state what was selected, what was confirmed, what can resume, and what can recover. An operator can select a policy that weakens part of the strict contract, but the system must identify that change before processing starts.
This contract is the difference between moving data during a successful demonstration and operating a transaction log processor through real failures.
Database-agnostic does not mean database-indifferent
PostgreSQL is the first implemented Remac source. MySQL, MongoDB, and Oracle are planned.
Each source integration must understand its database’s native log, position model, transaction behavior, type system, configuration, and recovery rules. A database-agnostic processor cannot erase those facts.
The shared part belongs inside the system:
- A position can be compared and stored.
- A selected change has an operation and source context.
- A sink confirms work under a declared policy.
- Checkpoint progress cannot move beyond incomplete required work.
- Recovery starts from a known safe position.
The database-specific part remains at the edge. PostgreSQL uses WAL, LSNs, replication slots, publications, and pgoutput. Another engine brings different mechanisms. The Remac Contract stays the same.
This design lets the product add source integrations without reducing every database to a weak common model.
One engine does not mean one destination
A committed change can become useful in several places.
An application service can react to a real-time event. A search system can update its index. An analytics system can receive current operational data. An audit archive can preserve selected changes and context. A database can converge with a source database. A recovery process can reconstruct declared state.
These are possible uses, not customer case studies. The complete result still depends on the selected source, sink, policy, configuration, and downstream system.
Remac runs as one self-contained processing engine in the customer’s environment. It does not require Kafka Connect, a JVM, or a Kubernetes cluster for its own processing path. Kafka and other existing systems can remain destinations when the workload needs them.
The goal is not to replace every data system. Remac owns the infrastructure between a committed database change and the systems that need that fact. Context creation, decisions, and actions remain outside the product.
The question that remains
The early prototype asked whether committed PostgreSQL changes could reach another system through a small, direct path. The answer was yes.
Remac asks the larger question:
How much useful work can one transaction log support when the processor treats correctness, recovery, performance, and operational simplicity as one engineering problem?
That question now covers logical and physical processing, six functions, multiple destinations, and one contract. The first JSONL file was only the beginning.
Remac is the high-performance, database-agnostic transaction log processor that grew from that work.
From commit to action.
Further reading