A payroll pipeline with a retirement date
Across five project sites, more than three thousand employees badge in and out on biometric access-control devices. Each generates a couple of events a day, and every one of them has to reach SAP, because SAP is the system of record that drives payroll.
Attendance was reaching SAP through four scheduled jobs owned by the access-control vendor and running on that vendor's servers at each location: a punch push, an employee pull, a shift pull, and a device health check. When a newer device fleet arrived, an in-house shim copied its events into the old vendor's database so those jobs could carry on feeding SAP.
The business then decided to decommission the vendor entirely. That decision, taken for sound commercial reasons, would have taken every piece of SAP connectivity down with it, at every site, without any single person being obviously responsible for noticing.
The old pipeline could not simply be rebuilt as it was
Reproducing the jobs that were being retired would have carried four properties forward that were already causing quiet damage.
- Failures were silent. A failed push into SAP — a network glitch, an SAP outage, an expired credential — meant those records were simply never sent. There was no retry queue, no failure list and no alert.
- Nothing tracked the state of a record. No individual event was ever marked as delivered, so when a job did not run, nobody found out until payroll noticed a discrepancy at the end of the cycle.
- Configuration lived in code. Device identifiers, endpoints, filters and credentials were compiled in and redeployed by hand at every location.
- The business rules were undocumented. Night-shift attribution and entry-exit classification existed only inside legacy scripts that nobody had read in years, and were understood by nobody currently employed.
The cost of all this was not primarily the engineering time. It was that attendance disputes had no authoritative record to appeal to. When an employee contested a figure, there was no way to reconstruct what the system had done or why.
What the environment fixed before we started
- Payroll could not be interrupted. The changeover had to happen while the business ran, against a fixed decommissioning date it did not control.
- The devices could not change. Whatever was built had to read from the existing device databases exactly as they were, with no modification at the edge.
- Sites are remote and connectivity is not guaranteed. Power interruptions and network partitions at a project site are ordinary events, not exceptional ones.
- SAP is the system of record and is not negotiable. Delivery had to go through its OData interface, with its authentication and CSRF requirements, on its terms.
- The HR-operations team would run it. Not engineers. Whatever monitoring, backfill and correction capability existed had to be usable by the people who own the payroll process.
Write it down before you try to send it
The replacement is a single Kotlin service that installs itself as a Windows service, with the Java runtime bundled, and takes over every SAP-facing responsibility on the day it goes live. The devices carry on capturing events exactly as before. Nothing changes at the edge.
The outbox comes before the network
Every event the service reads is written into a local SQLite ledger before any attempt is made to send it anywhere. Dispatch is a separate step that reads from that ledger, marks the state of each record individually, and never re-sends anything SAP has already acknowledged.
This single property is what makes the rest of the system tractable. A power cut at a remote site, an SAP outage in the middle of a batch, or a service restart during patching costs nothing, because the record was durable before the risky part began. The next run resumes exactly where the last one stopped.
Retries, and then a dead-letter queue
Transient failures retry automatically with exponential backoff, which absorbs the overwhelming majority of what goes wrong in practice. A failure that is genuinely permanent lands in a dead-letter state rather than being retried indefinitely or dropped, and an operator can requeue it from the console once the underlying cause is fixed.
A deliberate re-run is not an accidental retry
Every event is keyed so that a repeat delivery is recognised and discarded. That alone would make the system safe and useless, because there are legitimate reasons to send a record again: a shift schedule was corrected, a classification rule changed, a backfill is needed after an outage. So the service distinguishes between the two. An accidental retry is deduplicated away; an intentional re-run initiated by an operator is honoured, and audited.
Recovering the business rules
The rules buried in the legacy scripts were reconstructed and reimplemented test-first, which was the only way to be confident they had been understood correctly rather than merely copied. An event at two in the morning belongs to the previous day's shift, not the one that has just begun. Consecutive scans within a short window are one event rather than several. Entries, exits and breaks are classified rather than recorded as undifferentiated timestamps. Each site has an allowlist of the locations it may report.
None of that is glamorous, and all of it is the reason the figures reaching payroll are correct.
How the service is put together
Configuration instead of redeployment
Schedules, endpoints, credentials, batch sizes and location codes live in typed configuration rather than in the source. One artifact is deployed to all five sites and each is configured rather than rebuilt. The same design means the group's planned consolidation onto a single central deployment is a configuration change rather than a second project.
Parallel run first, cut over on evidence
A replacement for a payroll pipeline cannot be validated by testing. It has to be proven against the system it is replacing, on live data, before anybody is asked to trust it.
So the new service was brought up alongside the legacy jobs and run in parallel, reading the same devices and producing the same deliveries, with the two sets of results compared. Discrepancies were investigated as defects in our understanding of the business rules — which is what most of them turned out to be — rather than as noise. That period is what recovered the undocumented night-shift and classification logic accurately, because the legacy system was still there to disagree with us.
Only once the two agreed was the legacy pipeline switched off, site by site rather than at once. The parallel run continues at the client's discretion, which costs almost nothing and means the decision to fully retire the old jobs is theirs to take on evidence rather than on a date.
The console was built early rather than last, because the HR-operations team needed to be able to see what the new service was doing during the parallel run in order to develop any confidence in it at all.
What changed
The service runs unattended across five project sites, covering more than three thousand employees, and has replaced four vendor-owned SAP jobs with a single deployable artifact. The vendor was decommissioned on schedule without payroll losing its data supply.
Nothing that SAP has acknowledged is delivered twice, and nothing that was captured is lost, because both properties are structural rather than operational. Every call into SAP is stored with its request, its response and its duration, so when payroll asks why a particular figure looks the way it does, the answer can be produced from the console rather than reconstructed from memory. Attendance disputes now have an authoritative record to appeal to, which they did not before.
Operators can rewind a source's starting watermark, backfill a date range after an outage, and browse ingested data read-only, with every one of those actions audited. The system runs on the client's infrastructure and they own all of it.
What it is built on
SQLite was chosen deliberately over a server database: the outbox has to survive a power cut at a remote site with no database administrator present, which is a requirement an embedded file-backed store meets and a networked one does not.
Lessons
The most valuable engineering decision on this project was also the least sophisticated: write the record down before attempting the network call. Everything else — the retries, the dead-letter queue, the backfills, the audit trail, the ability to run in parallel with the system it replaced — is only possible because the durable record exists first.
The second lesson concerns undocumented business rules, which are the real risk in any replacement of a legacy integration. They cannot be recovered by reading the old code, because the old code is a statement of what happens rather than what was intended. They are recovered by running the two systems side by side and investigating every disagreement. That period is not a delay in the project; it is the project.
The third is that an integration is only finished when somebody who is not an engineer can answer a question about it. Payroll disputes are a normal part of operating a workforce. The system that resolves them has to be operable by the team that fields them.