Case study · live production system
A live production platform, engineered end to end
A system RTN designed, deployed, and operates. Domain specifics are generalized; the engineering is described exactly as built. Figures as of 2026-07-26.
773unique users served
235active in the last 7 days
~2,365usage hours delivered
2,681sessions
The user base more than doubled in 13 days. After re-architecting from a single in-house host to a three-node containerized Docker fleet over 2026-07-12 to 2026-07-13, the base grew from 377 at cutover to 773, with every single record preserved and zero data loss.
What it is. RTN runs a live, always-on production platform that serves end users continuously from a self-managed cloud host, in continuous operation since 2026-05-20. It was migrated from a single in-house host to a three-node containerized Docker fleet (v1.x to the current v2.0.5): the migration began 2026-07-12, all three nodes were back in production by 2026-07-13, and every single record was preserved through the move with zero data loss.
The architecture. One cloud VPS (8 vCPU / 8 GB RAM) runs a Docker container fleet: three real-time application nodes, one shared database, and one update-only job. Each node is core-pinned and memory-capped, so a spike on one cannot starve the others. A single read-only application install is shared across all nodes, with a per-node writable overlay on top, and one update-only service is the sole writer to the shared install. Nodes boot in a serialized, health-gated order, so nothing starts until its dependency is genuinely up.
The custom service. Each node runs twelve service modules, three of them built in-house in C# and running on the .NET 10 runtime. Those three started as a single module and were split by responsibility as the platform matured: scheduling and gating, unified stats, and supporting features. The remaining nine are third-party, on the principle that custom code is written only where the value justifies maintaining it. All three in-house modules are kept in version lockstep with a single source-of-truth VERSION file.
One identity across the fleet. A single shared MariaDB database gives every user one identity across all three nodes. Usage stats, usage hours, and access-control history follow the user everywhere, written through an asynchronous write-behind flusher rather than blocking the live path. Application nodes never write back to the configuration source of truth.
How changes ship. A separate build and staging host is the single source of truth; changes flow one way to production and never the reverse. Deployments go per-service, one node at a time: pre-flight safety backup, transfer over SSH and rsync, restart, then verification by healthcheck and liveness probe before a guarded commit that refuses the main branch, never pushes, and adds no automated attribution. A secret-scrubbing publish gate strips real addresses, tokens, and identifiers before any artifact is ever made public.
Resilience and self-recovery. The platform recovers unattended. An ordered boot unit brings the full stack up in dependency order after any reboot, each service gated on the previous being genuinely up. A self-healing check repairs a broken install every fifteen minutes and refuses an unsafe action when disk is tight, raising an alert instead. If a runtime component fails to load, the affected node drops to a degraded fallback and stays reachable rather than going dark. System package holds keep an unattended host upgrade from pulling the container runtime out from under the running fleet. Each of these was proven on live reboots on 2026-07-17.
Operations and durability. A chat-ops surface driven by operational webhooks provides a self-updating live status card refreshed every minute, action and audit logs, and a consolidated update fan-out. Durability follows a 3-2-1 rule: an automated database dump to OneDrive every 24 hours at midnight (UTC-7), a pull to the build host, and a NAS/USB mirror, with a nightly prune of aged artifacts.
The operating principle. Solve the problem at the lowest stable layer, automate the surface above it, document the runbook below. The incident below is that principle in practice, reproduced rather than summarized.
Incident report, shown rather than told
SymptomA startup gate inside the platform's scheduling and gating service released on only 2 of roughly 250 session cycles over 11 days. Operators worked around it with 78 manual restarts before the pattern was caught.
First hypothesis (wrong)A timing calculation inside the gate.
Root causeEvery session cycle disconnects and reconnects all active sessions. An internal state dictionary the gate depended on was cleared on disconnect, and the event that normally repopulates it never re-fired on reconnect. The gate's live-state check read zero every time, so it never released.
Regression sourceBisected back to an earlier defensive-coding change.
FixRewrote the gate to poll live connection state directly on every tick instead of trusting a cached, event-driven dictionary, with a managed fallback and per-tick logging.
Status at deployment (2026-06-11)Fix deployed, rollback staged, verification pending the next full cycle.
Why it matters to a quality operation. The same discipline behind a quality system, proven on a system RTN runs itself: a controlled one-way change flow with pre-flight backup and post-change verification (change control), a guarded commit trail (configuration management), a single source-of-truth version (revision control), a scrub-gated release (controlled release), 3-2-1 backups that fail loudly (records retention and disaster recovery), and unattended self-recovery (business continuity).
Flow A · change to user
A change travels one way: source → pipeline → infrastructure → runtime → the user experience.
Flow B · usage to unified record
Live usage flows the other way: runtime → the shared data tier → one identity per user.
01Source of truth · build and staging host
A single build and staging host is the source of truth. Changes flow one way to production and never the reverse. Triple-redundant local backups.
edited and validated off-production, never live ↓
02Version control and publish gate
Separate source repositories with full history. A secret-scrubbing publish gate strips real addresses, tokens, and identifiers before any artifact is made public. A single VERSION file is kept in lockstep with the service-module version.
deploy per node, one at a time ↓
03Deployment pipeline
A pre-flight safety pass runs before every deploy: back up state, update, restore configuration, verify.
1
Edit
change on the master tree
2
Sync
fan out to derived artifacts
3
Build
compile the service module
4
Transfer
SSH + rsync, pre-flight backup
5
Verify
healthcheck + liveness probe
6
Commit
guarded: refuses main, never pushes
into each node's writable overlay ↓
04Infrastructure · the production host
One cloud VPS (8 vCPU / 8 GB). A Docker container fleet runs three application nodes, one shared database, and one update-only job. A single read-only application install is shared across nodes; each has its own writable overlay; the update-only service is the sole writer. Every node is core-pinned and memory-capped.
node A
core-pinned · capped
node B
core-pinned · capped
node C
core-pinned · capped
shared DB
unified identity
boot order: DB → A → B → Ceach waits on the last being genuinely up
if a component fails, the fleet recovers on its own ↓
RESILIENCESelf-recovery, unattendedProven on live reboots · 2026-07-17
An ordered boot unit restarts the full stack in dependency order after any reboot, each service gated on the previous being genuinely up. A self-healing check repairs a broken install every fifteen minutes and refuses unsafe actions when disk is tight, alerting instead. If a runtime component fails to load, the affected node drops to a degraded fallback and stays reachable rather than going dark. System package holds stop an unattended host upgrade from pulling the container runtime out from under the running fleet.
ordered boot unitself-healing update checkdegraded-mode fallbackpackage holds
inside each node · seed → patch → launch ↓
05Runtime · one representative node
Shared read-only install plus a writable overlay. Twelve service modules per node: three built in-house (C#, on the .NET 10 runtime) covering scheduling and gating, unified stats, and supporting features, plus nine third-party modules for access control and messaging.
usage writes stats, hours, and access records ↓
06Data tier · one identity across the fleet
A single shared MariaDB database gives every user one identity across all nodes. Usage stats, usage hours, and access-control history follow the user everywhere, written through an asynchronous write-behind flusher rather than blocking the live path. Nodes never write back to the configuration source of truth.
07Operations, observability, and durability
A chat-ops surface driven by operational webhooks provides a self-updating live status card refreshed every minute, action and audit logs, and a consolidated update fan-out. Durability follows a 3-2-1 rule: an automated database dump to OneDrive every 24 hours at midnight (UTC-7), a pull to the build host, and a NAS/USB mirror, with a nightly prune of aged artifacts.
