// Track

Infrastructure Drift

The Silent Failures Behind a Healthy Status Page

Ten hard-won lessons from operating a multi-machine homelab over Tailscale — merge gaps, lying health checks, exponential drift, permission bombs, ghost processes, network ambiguity, Docker caching traps, singleton enforcement, version observability, and building automated drift detection. Every lesson draws from a real incident.

10 lessons~88 min total

Lessons are shown in recommended order. Complete them in sequence for the best experience — or jump to any lesson.

Lesson 232·8 min read

Code Merge Is Not DeploymentNew

PR merges don't deploy themselves. How code-to-runtime gaps create silent outages across distributed infrastructure.

Lesson 233·8 min read

Health Checks LieNew

A container can report 'healthy' while missing critical API endpoints. Liveness is not correctness.

Lesson 234·9 min read

Drift Is ExponentialNew

One skipped deploy becomes two, becomes 'everything is 5 commits behind.' Drift compounds silently across machines and repos.

Lesson 235·9 min read

Permission Drift from AutomationNew

AI agent sessions create files with restrictive permissions that silently break Docker builds. The invisible bomb pattern.

Lesson 236·9 min read

Ghost Processes AccumulateNew

Zombie processes, orphaned test runners, and duplicate instances pile up without active cleanup. They are invisible until they aren't.

Lesson 237·8 min read

Two Instances, One PortNew

The same port on different network interfaces can serve completely different data. Trust but verify which instance you are actually hitting.

Lesson 238·8 min read

Docker Cache Hides FixesNew

When COPY is cached, your permission fix or code update doesn't take. Understanding Docker layer caching saves hours of debugging.

Lesson 239·9 min read

Duplicate Services Equal Financial RiskNew

For trading bots and stateful services, duplicate instances are not waste — they are active danger. Singleton patterns are non-negotiable.

Lesson 240·8 min read

Version ObservabilityNew

If you cannot tell what version is running without SSH and git log, you have no observability. Embed version info everywhere.

Lesson 241·12 min read

Building a Drift Detection SystemNew

From manual SSH checks to automated deployment sync. How to build a system that catches drift before your users do.