01 · Context
Real load changes the questions
In a lab, throughput tends to dominate the discussion. In production, predictability and the ability to explain system behavior matter just as much as raw speed.
- External integrations with latency and error rates outside our control.
- An administrative area with cookie-based sessions and explicit permissions.
- Enough observability to diagnose causes instead of symptoms.
- Hybrid caching that lowers cost without turning Redis into a single point of failure.
02 · Architecture
Limits before scale
The bootstrap defines the operating envelope: pools, timeouts, concurrency, and health checks are configured before the first request is accepted.
- GOMAXPROCS and garbage collection tuned against real CPU and memory profiles.
- A PostgreSQL pool with explicit open, idle, and lifetime limits.
- An HTTP server with finite body limits, buffers, and timeouts.
- Readiness gates that keep an instance out of rotation until essential dependencies are ready.
An infinite queue is not resilience. Under a spike, rejecting early and recovering quickly beats accumulating work until the process collapses.
03 · Performance
Keep the hot path simple
Most gains came from removing work, controlling allocations, and preventing slow dependencies from holding resources indefinitely.
- Efficient serialization with validation at the boundary and reusable buffers.
- Shared HTTP clients, connection pooling, and timeouts for each stage.
- Compression and ETag only where total cost justified the benefit.
- Per-route profiling so optimization followed production evidence.
04 · Caching
Caching is an availability strategy
Redis shortens the read path, but it should not single-handedly control API availability. The design includes a bounded local fallback and TTLs aligned with data volatility.
- Redis as the primary layer, with explicit TTL and invalidation policies.
- A bounded local cache for essential responses during instability.
- Versioned keys and hit, miss, error, and latency metrics.
- Predictable eviction to prevent silent memory growth.
When Redis fails, the platform should lose capacity—not disappear.
05 · Observability
Metrics must answer questions
Dashboards were designed around operational decisions: should we scale, limit, roll back, or investigate a dependency?
- Latency histograms and outcome counters by normalized route.
- Request and response sizes to detect abnormal payloads.
- Runtime signals: CPU, heap, goroutines, and GC pauses.
- Impact-oriented SLOs and alerts that avoid unactionable noise.
Normalizing route parameters prevented unbounded cardinality and kept Prometheus usable during incidents.
06 · Security
Controls close to the boundary
The administrative area uses protected sessions and explicit RBAC. Authorization remains on the server, close to the operation it protects.
- HttpOnly, Secure, and SameSite cookies matched to the flow.
- Declarative permissions and an audit trail for sensitive operations.
- Payload limits and validation before business logic.
- Diagnostic tools exposed only in controlled environments.
07 · Incidents
The mistakes that changed the design
The platform's strongest patterns came from observed failures. Every incident became a verifiable system change, not just documentation.
- Metric cardinality led to mandatory label normalization.
- Total Redis dependency led to a bounded local fallback.
- Unbudgeted retries led to backoff, jitter, and attempt limits.
- Excessive logging led to sampling and environment-aware levels.
08 · Checklist
What I would verify before the next deploy
Sustainable performance is a property of the whole system, and automation must protect it.
- Timeouts, pools, and body limits covered by contract tests.
- Caching with verifiable fallback, TTL, eviction, and telemetry.
- Normalized routes and alerts linked to clear SLOs.
- Readiness, rollback, and representative load testing before release.