The traditional way to migrate a large commerce database goes like this. Pick an ETL tool. Get a licence. Get the platform team to provision it. Get the DBAs to open firewall rules. Get a consultant who knows the tool. Hold a weekly meeting about the mapping spreadsheet. Weeks later you have a pipeline, and its logic lives inside a GUI nobody can diff.

I did a recent migration a different way. One engineer, an AI assistant, and a Jenkins job that starts a pod. The first working end-to-end version took two days.

This post is about what that means, and what it doesn’t. I’m going to argue that this is the future: custom software, built on demand, for a single bounded problem, is becoming the default. I’m also going to show you the bug that proved the AI didn’t remove the need for engineering judgment. I think that’s the more important half.

The problem

Move order templates (saved shopping lists, plus their sharing rules, file uploads and audit history) out of a legacy Oracle commerce platform and into a modern service backed by Azure SQL. The source model was 18 tables for the core domain, 6 for file uploads and 7 for audit events. The target was 6, 2 and 2 tables with a new schema, new UUID keys, and different rules for who can see what.

The biggest target table had 9.5 million rows.

What I built

A batch ETL pipeline, where each phase is an independent Kubernetes Job and blob storage is the checkpoint between phases:

Jenkins "Build with Parameters"
   └─► Helm deploys a Job pod into the target cluster
         └─► python migrate.py <flags>

 EXTRACT    Oracle ──(4 parallel workers, 50K-row batches)──► Parquet in blob
 TRANSFORM  Parquet ──(DuckDB, in-process, plain SQL)──────► Parquet in blob
 LOAD       Parquet ──(BCP bulk insert / MERGE)────────────► Azure SQL

Some choices that made it small and sturdy:

  • Parquet at every stage. Extract writes one Parquet file per source table. Transform reads them and writes one per target table. Any phase can be re-run without repeating the earlier ones, and every intermediate state is a file I can open and query.
  • DuckDB as the transform engine. No database server to run. The transform is SQL over Parquet files, in-process. Reviewable, diffable, testable.
  • Deterministic IDs. The legacy system had string IDs like gl0000000001. The new system uses UUIDs. I derive each UUID from an MD5 of <entity-prefix>:<legacy-id>, so the same legacy record always maps to the same UUID. Loads are idempotent and diffs are stable.
  • Different load strategies per data shape. Domain tables change over time, so they go BCP → staging table → MERGE. Audit events are immutable, so it’s a client-side anti-join followed by a direct bulk insert. Small change sets use plain INSERT/UPDATE/DELETE. One child table needs an IDENTITY foreign key, so it falls back to row-wise loading. One size does not fit all.
  • Index management around bulk loads. Drop non-clustered indexes, load, rebuild. With a --rebuild-indexes command for when a job gets killed midway.

I won’t walk through every detail here. The point is that none of it is exotic. It’s ordinary, competent data engineering, assembled in days instead of weeks, because the AI wrote the scaffolding, the SQL and the glue while I made the design calls.

Then it stopped being a migration

Here’s the part that changed my mind about the whole category.

Once the extracts were Parquet files on blob storage, the migration pipeline turned into a local analytical dataset. Questions like “which orders did this supplier receive in the last N days?” or “how many templates are shared across organisations?” no longer needed a query against the live production database, a DBA, or a read replica. I asked them against Parquet with DuckDB on my machine.

That’s the same idea as pairing an event stream with a periodic diff: a cheap, immutable snapshot you can interrogate freely beats a live system you have to be careful with. The snapshot also takes read load off the transactional database, which, as I wrote in Internal Workflows Deserve Their Own Stack, is where a lot of avoidable production incidents come from.

And it extended into continuous sync. The pattern:

  1. First run: full sync from Oracle to Azure SQL.
  2. Every run after: still extract everything from Oracle, transform it, then diff the new transformed output against the previous run’s output. Inserts, updates and deletes fall out of three joins (ANTI JOIN for inserts and deletes, SEMI JOIN plus EXCEPT for updates).
  3. Apply only the delta.

Extracting everything every time sounds wasteful. It’s actually the point: the source system doesn’t need change tracking, triggers or CDC. I didn’t touch the legacy database’s schema or ask anyone’s permission to. The diff is computed on the transformed output, so a change in my own mapping logic also shows up as a delta, and there’s only one diff strategy to maintain.

Why I think this is the future

The economics of software used to push you toward platforms. Building something bespoke cost weeks of engineer time, and an ETL suite, a low-code tool or a vendor was cheaper than that. The tradeoff was real, even if the tools were clumsy: you bought a generic thing and bent your problem to fit it.

AI changes one side of that equation. If the first version of a purpose-built tool takes two days instead of weeks, then for a bounded problem the calculation flips:

Platform toolCustom, built on demand
Time to first resultWeeks (procurement, setup, coordination)Days
Fit to the problemYou adapt to the toolThe tool is shaped like the problem
Logic lives inA GUI or vendor formatGit: SQL, Python, a Dockerfile
Reviewable, diffable, testableHardNormal engineering practice
Teams to coordinateSeveralAlmost none
Throwaway when doneRarelyYes, which is fine

The last row matters most. When building is cheap, software no longer has to justify a long life. You can write a tool for one migration, one analysis, one reconciliation, and delete it afterward. The threshold for “worth building” drops dramatically.

I’d call it custom software as a service, where the service is you plus an assistant: bespoke tooling on demand, for the question you have today.

What the AI did not do

Now the important half. A single engineer plus AI built a pipeline quickly. It was not correct the first time, and the problems that remained were the kind AI doesn’t flag on its own.

1. The wrong UUID function

The transform has two helpers that both turn a legacy ID into a UUID. One uses MD5, the other uses Python’s uuid5 (SHA-1, namespaced). They produce completely different UUIDs for the same input. The database held the MD5 ones. A different pipeline (file uploads) used the other helper to reference a list ID inside a message column, so every reference pointed at a UUID that didn’t exist.

Both functions looked reasonable. Both returned a valid-looking UUID. Nothing failed. The data was just quietly wrong. It took a human noticing that the IDs didn’t match. I now keep a loud comment above those helpers, and I’d suggest the same for any code with two “obviously equivalent” ways of doing a critical thing.

2. A delta baseline that trusted the wrong source of truth

This is the one I’d put on a poster.

The delta pipeline treats the blob-stored output from the previous run as “what’s already in the database”. It never verifies that against the database itself. The sequence on each run was:

transform  →  upload new output/ to blob  →  load into SQL  →  promote output to "previous"

Look at the order. The new output is uploaded to blob before the load runs. If the load fails halfway, the earlier batches are already committed, the later ones are not, and the run raises an error. The “promote to previous” step never happens, but the blob already holds the new state.

The next run downloads that blob as its baseline, recomputes, and finds nothing to do for the rows that never got committed. They look identical to the baseline, so the delta says “no change”. No error, ever, and they’re never retried.

It really happened. A run computed about 6,400 new sharing rows, applied the deletes and the parent inserts, and then crashed on a foreign-key violation from a data edge case partway through the child inserts. The edge case was fixed afterward. The next run, as designed, found almost no difference, and one valid child row stayed missing from the database. A silent, permanent gap.

How I found it, and how it can be repaired:

  • My validation script checks the internal consistency of the Parquet files. It has no visibility into the real database, so it could never see this class of bug.
  • The one tool that diffs the live database against the baseline (an export-and-compare command) is what exposed it.
  • A full load heals it, because MERGE ... WHEN NOT MATCHED THEN INSERT reconciles against what’s actually in the table, not against a stored baseline.

The fix I’ve noted but not yet implemented is straightforward: upload the new output (or promote it to baseline) only after the load fully succeeds, and add a periodic compare against the real database.

I want to be honest about this. It’s a design flaw in a pipeline I would have called “robust” before this, and the AI would have happily reproduced it a hundred times. Checkpointing against your own artefacts instead of against reality is a classic trap. The lesson I take from it is the same one I took from the event-stream post: any sync that trusts its own record of what it did will drift, and you need an independent check against the real system.

3. Domain knowledge

Whole categories of rules weren’t in any table definition. Organisation-level lists deliberately have no “sold-to” account in the legacy system, because it was resolved from the user’s session at runtime. Some lists are referenced by audit events after being hard-deleted from the source, so the transform reconstructs “ghost” stubs. Who counts as a locked-by user depends on a combination of privileges.

An assistant can write the SQL once you tell it these rules. It can’t discover them. That part is still reading old code, asking the right question and knowing what the business meant.

What changes about the job

If I put the whole experience on a scale:

AI made fasterAI did not change
Boilerplate, scaffolding, glue codeDeciding what “correct” means
SQL for each of the target tablesFinding the rules that aren’t written down
Bulk-load plumbing, Dockerfile, Helm, Jenkins wiringFailure-mode thinking: what happens when step 3 of 5 fails?
Exploring ideas (three load strategies, tried and compared)Verifying against the real system, not your own output
Documentation, once I explained the designOwning it in production

That last row is the real tension with “custom software on demand”. Cheap to build isn’t the same as cheap to own. A pipeline you wrote in two days still needs someone who understands it at 3 a.m. when a load fails. Bespoke software has a bus factor of one by default. The countermeasure is to make the artefacts legible (files, SQL, an architecture doc, runnable commands for re-driving any phase) so that “someone else” is a reasonable ask.

When I’d reach for this, and when I wouldn’t

I’d build it myself when:

  • The problem is bounded: a migration, a reconciliation, an analysis, a one-off integration.
  • The tool can be thrown away or rebuilt when the need changes.
  • The data and rules are unusual enough that a generic tool would need more bending than a fresh program.
  • I can verify the result independently against the source and the target.

I’d still buy or adopt a platform when:

  • It’s long-lived, shared infrastructure with many users, audit requirements or an SLA nobody wants to own.
  • There’s a mature, supported product that already fits the problem without bending.
  • I couldn’t realistically verify my own output.

The takeaway

The thing that changed isn’t that engineers got replaced. It’s that the cost of the first working version of a bespoke tool dropped by an order of magnitude, and with it, the bar for building one. A migration that would have been a multi-week programme with a vendor tool and several teams became a two-day build by one person, and then grew into a reusable analysis environment as a bonus.

The thing that didn’t change is that someone has to be right about the hard parts: the failure ordering, the identity of a record, the difference between “the baseline says it’s loaded” and “it’s loaded”. The AI got me to a working pipeline quickly. Getting it to a trustworthy one was still my job. Judging by the bugs above, it will be for a while.