The estate
Content had been accumulating for years, and it had accumulated in five different places. Some sat in ordinary directories on local disks. Some lived on Windows network shares that had been added department by department, each with its own conventions. The rest was held in three document management systems, every one with its own storage model, its own metadata scheme, and its own idea of what a version is: an Arabdox archive, a Constellio export of XML and blob storage, and a database-backed DMS.
Together they held more than a million files and folders, roughly three terabytes in total. The material mattered. Student and staff records, correspondence, and scanned documents that an auditor can ask to see years after the person who filed them has left. All of it was headed for OpenText Content Server as the single system of record.
A migration of this length will be interrupted
The difficulty in moving three terabytes is not bandwidth. It is that a job running for days will be interrupted, and the interruption is not hypothetical. A network share disappears for ninety seconds. A disk fills. A server reboots overnight for patching. An operator stops for a maintenance window and resumes in the morning.
When that happens, a migration has two ways to fail and both are unacceptable to this organization. It can lose a document, which nobody discovers until an auditor asks for it years later. Or it can upload the same document twice, leaving the target with duplicates that are harder to clean up afterwards than the original migration was to run.
The approach most teams reach for makes both outcomes likely. A script that scans and uploads in a single pass holds its progress in memory, so a crash at hour sixty leaves starting over as the safest available option — and starting over on a partially completed migration is exactly how duplicates are created.
What the environment fixed before we started
- The source systems were read-only and irreplaceable. Nothing in the migration could modify, lock or degrade a live repository, several of which were still in daily use during the move.
- Structure had to survive, not just content. Folder hierarchies, multi-version documents and source metadata all had to arrive intact. A migration that moved bytes and dropped context would have failed its purpose.
- The target could not hold everything the sources recorded. Legacy categories, authors and dates existed in fields Content Server has no native equivalent for.
- It had to run inside the client's network, on machines administered by people who should not have to install and version-manage a Java runtime in order to move their own documents.
- Analysts, not engineers, would operate it. The team responsible for the content had to be able to define a job, review what was found, run the transfer and retry failures without us on the call.
An engine in which the unsafe outcomes are unreachable
The engine was designed so that losing a document and duplicating one are not merely unlikely if everybody is careful, but structurally unavailable. Three properties do that work.
Scanning and transfer never share a fate. A Scanner walks the source and writes an inventory of what it finds into PostgreSQL. It is read-only on the source and never touches the target. A separate Worker reads the records marked ready and uploads them. It never goes back to the source. Between them sits the database, and the database is the only thing that knows what has happened. Nothing of consequence is held in an in-memory queue: every discovered node, every version status and every checkpoint is committed before the next step begins. A hard kill costs the request that was in flight and nothing else.
Uploads are idempotent. Every document carries a stable identity derived from the source — a source UID combined with a checksum of the content. Before uploading, the Worker checks that identity against the target rather than trusting a local flag claiming the work was done. Re-running a completed batch is therefore safe by construction, and an operator who is unsure whether a job finished can simply run it again.
Uploads are verified. A successful HTTP response is not evidence that a document arrived. After upload, the engine reads the document back out of OpenText and confirms it is present and intact before marking the record complete. The migration is finished when the target says so, not when the network does.
Preserving what the target could not hold
Multi-version documents keep their full history: the most recent version becomes the canonical document and earlier ones are archived alongside it. Folder hierarchies are mirrored exactly. Where the legacy systems recorded metadata that Content Server has no field for, the engine writes it into a JSON sidecar attached to the document, so nothing is lost even where the target cannot represent it natively. Filenames are sanitised for characters the operating system will not accept, with collision-safe renaming.
How the pipeline is put together
The engine ships as a single Windows executable: Spring Boot backend, React interface and an embedded PostgreSQL, with the Java runtime bundled inside it. There are no prerequisites to satisfy on the machine it runs on and no build server to depend on.
Rehearsed to local disk before it touched production
The engine ships with a filesystem target connector alongside the OpenText one. It runs the entire pipeline against local disk instead of Content Server: the same scanning, the same checkpointing, the same idempotency, the same reports. The team could rehearse a complete migration end to end, inspect exactly what would land where, and only then point the same engine at the real target. A migration that has been performed once against a harmless target is a substantially different risk from one that has only been reasoned about.
The control room enforces the same discipline in normal operation. An analyst defines a job, runs the scan, and browses the discovered inventory record by record before anything moves. Nothing is transferred until a person has looked at what was found and agreed that it is right.
Changes were branch-gated through feature, development and integration environments before reaching the main line, with a scripted harness that validated connectivity against each source and ran the suite before any promotion. Execution history records every scan and transfer run with its status, its timing and its checkpoint, which is the audit trail that answers the question that always arrives halfway through a migration: what ran, when, and how far did it get?
What changed
Five legacy sources are connected and retiring. Roughly three terabytes and more than a million files and folders have been consolidated into OpenText Content Server with folder structure, version history and source metadata preserved. No document has been lost, and none has been written to the target twice — not because the migration was watched carefully, but because neither outcome is reachable in the design.
The engine is live at the client's site and their own analysts operate it without us. The code, the database and the control room belong to them, running in their environment, with no dependency on the team that built it.
What it is built on
The stack was chosen for the environment rather than for preference: a single artifact the client's own administrators could install without prerequisites, an embedded database so there was no server to provision, and a browser interface so an analyst could operate a migration without a terminal.
Lessons
The engineering that mattered here was not clever. It was a refusal to hold important state anywhere except a database, and a willingness to pay the cost of that discipline on every single record. That decision is invisible in a demonstration and decisive at hour sixty.
The second lesson is about who operates a system. It would have been faster to build a command-line tool and run the migration ourselves. It would also have left the client dependent on us for every subsequent source, every re-run and every correction. Building the control room cost weeks and bought the client the ability to retire their next legacy system without a supplier.
The third is that verification is not paranoia. Reading each document back out of the target after writing it roughly doubles the number of calls the engine makes, and it is the only reason anybody can state, as a fact rather than a hope, that nothing was lost.