Hypergrowth storage needs predictable work, not unlimited queries

What happened: a shared library became a control plane

Habitat began in 2024 as a Python library over Azure Cosmos DB. It hid schema lookup, routing, authorisation, encryption, serialisation, request shaping and connection pooling from product teams. As products and services multiplied, distributing new routing logic and fixes through every client became slow and brittle.

OpenAI moved that logic into a standalone service. The change created one deployment and observability point, and one place to enforce access controls, audit activity and limit direct access to underlying storage. It also added network and compute overhead to an already latency-sensitive path.

The team accepted Python's short-term inefficiency to stabilise the platform and its APIs, then rewrote the service in Rust when scale justified it. OpenAI reports that the Rust service now handles 95% of production requests and is six times more CPU-efficient and 15 times more memory-efficient than the Python version.

Why it matters: average health can hide the failure users feel

A product request may depend on hundreds of database calls, so the slowest call shapes the experience. OpenAI found that Python's asynchronous event loop could stall ready work behind CPU-heavy routing, encryption, compression and background tasks even when downstream storage had responded quickly. CPU and memory dashboards alone did not explain that tail latency.

Connection reuse introduced another subtle failure. A last-in-first-out pool repeatedly sent work back to slower processes after a burst, concentrating traffic on instances already struggling. Switching to first-in-first-out reuse broke the feedback loop; service-aware balancing later reduced reliance on client behaviour.

Scaling out many lightweight processes created a different risk: connection churn could overwhelm dependencies. OpenAI used Envoy, HTTP/2 multiplexing, rate limits and circuit breakers to reduce the connection load and contain downstream pressure.

The practical implication: make expensive work explicit

Centralise cross-cutting storage behaviour when coordinated client rollouts become an operational hazard, but treat the new service as a critical dependency. Define availability and latency objectives, isolate tenants, rehearse regional failure and preserve an escape path before directing every product through it.

Measure queueing and scheduling delay alongside utilisation, saturation and downstream latency. Add jitter to periodic work, cap concurrency from observed tail behaviour and test connection-pool choices under burst and partial-degradation conditions. A balanced average is not proof that the slowest workers are healthy.

Keep online operations bounded. Habitat deliberately offers a constrained NoSQL-style API and moves complex analytical reads to isolated secondary systems. For other platforms, the equivalent is to require indexed, predictable requests on the serving path; put scans, joins and high-fan-out exploration behind separate capacity, budgets and failure boundaries.

If the team deliberately accepts technical debt to move faster, record the reason, expected limit and exit signal at the start. OpenAI's sequence worked because Python bought time for API maturity and was later replaced. Without a measurable trigger and funded migration path, a strategic compromise simply becomes permanent risk.