Lessons-Learned

Event-driven sync drifts. Pair it with a cheap periodic dump-and-diff and you get reconciliation almost for free.

I cabled my five-node rack behind a switch to get off wifi. The cutover worked. Internet bandwidth dropped 50-85% across the cluster. Here is what happened and why I am keeping it anyway.

Two HP worker nodes dropped off my cluster after I pulled their USB wifi adapters. When I plugged them back in, neither came up. The adapters were fine – I’d put them in the wrong machines.

Validation alerts worked in dev but showed wrong messages in test. The culprit: an internal analytics library had silently overridden the native Error constructor.

A simple SELECT by primary key took 6 seconds. Postgres was fine. The network was fine. The culprit was a default Linux wifi setting I’d never heard of.

In the ATG monolith, production DB credentials lived behind a JBoss SSH gate — effectively unreachable without infra access. After modernizing to microservices on Azure Kubernetes, every developer’s Azure account could read prod DB, Redis, and Service Bus secrets from Key Vault with a single CLI command. VPN and device whitelisting gated the network path, but not the humans. The migration didn’t just change our architecture — it quietly widened the insider blast radius.

Oracle ATG Commerce is legacy now, but its Nucleus component model, immutable repository items, and config layering were ahead of their time. Its distributed locking, custom messaging, and datasource switching were not. A reflection on what held up and what didn’t.