Scaling Infrastructure and Intelligence: OpenAI’s Storage Solutions and the New Decisions API

As applications, agents, and artificial intelligence models scale to serve billions of users globally, engineering teams face monumental challenges in backend architecture, data management, and response latency. OpenAI recently shared deep insights into two major technical pillars: scaling its online storage platform, Habitat, to serve over one billion ChatGPT users, and launching the general availability of the Decisions API, designed for fast, focused, low-cost application workflows.

---

Rapidly Scaling Online Storage: Habitat and Azure Cosmos DB (Part II)

Serving over 1 billion users across 40 geographic regions requires an online storage platform that can respond in milliseconds. OpenAI's storage platform, Habitat, manages over 507 petabytes of data. While Part I of OpenAI's engineering series covered Habitat's evolution into a robust service, Part II focuses on scaling its underlying storage engine: Azure Cosmos DB.

Overcoming Multi-Tenancy Challenges

As infrastructure grows, teams often transition from single-tenant systems—which limit the blast radius but suffer from over-provisioning—to multi-tenant architectures. Habitat's journey with Azure Cosmos DB highlighted several critical scaling hurdles:

• Logical Partition Key Limits: Early API choices exposed logical partition keys (like user_id or workspace_id) directly to clients, causing heavy collisions and constraining overall throughput.
• Noisy Neighbors: Because resource limits are enforced at the physical partition level, heavy workloads on niche features could disrupt core login and signup flows.
• Weak Provisions & Splits: As data volumes grew, physical partitions split, automatically reducing the effective per-partition throughput limit even when user demand remained stable.

The Solution: Co-Managed Services and Improved Serverless Modes

OpenAI collaborated closely with the Azure Cosmos DB team to engineer a sustainable shared-tenancy design:

• Hot Key Throttling: Automatically throttles the hottest keys first to prevent a single noisy neighbor from monopolizing physical partition resources.
• Quality of Service (QoS): Assigns lower priorities to offline workloads (such as bulk data extracts) so interactive user requests always take precedence.
• Enhanced Serverless Mode: Custom updates allowed physical partitions to scale up to 10K RU/s while introducing RU-usage-based splits alongside storage-based splits.

Global Performance Optimization

To drive read p50 latency down from roughly 120 ms to 10 ms (a 12x improvement), OpenAI implemented a multi-layered read strategy:

1. Integrated Lookaside Caching: Utilizing Valkey servers close to processing regions, combined with Change Data Capture (CDC) via Azure Cosmos DB's change feed, to propagate global cache invalidations within seconds.
2. Rust Refactoring: Rewriting proxy layers in Rust to eliminate Python-related CPU overhead.

---

Building Responsive Applications with the Decisions API

Applications and AI agents routinely make micro-decisions—such as classifying inputs, routing requests, or determining the next step in a workflow. While simple individually, their latency and costs add up quickly at scale. Built with GPT-6 Luna, the Decisions API is now generally available to help applications make near real-time choices from text, images, and structured inputs up to 10x faster than standard completion endpoints.

Supported Answer Types

The Decisions API supports three distinct answer formats to fit various business and technical logic:

| Answer Type | Description | Main Result | Use Case Example | | :--- | :--- | :--- | :--- | | predicate | Estimate the probability that a condition is true. | A number from 0 to 1. | Determining if uploaded product photos show visible damage. | | choice | Select one option from a predefined list. | The chosen value (includes confidence probabilities). | Deciding whether an agent should continue, retry, or ask for clarification. | | score | Rate an input against ordered levels in a rubric. | Probability-weighted average of level indices. | Assessing customer feedback severity or document categorization. |

Real-Time Voice Experiences with GPT-Live

The Decisions API natively integrates with GPT-Live, allowing voice applications to make split-second decisions during live conversations. For instance, customer support bots can dynamically decide whether to escalate an issue, or interactive digital avatars can adjust their facial expressions (such as smiling or looking surprised) based on streaming transcript updates.

{
  "type": "decisions",
  "event_type": "session.input_transcript.delta",
  "decisions": {
    "questions": [
      {
        "type": "choice",
        "name": "reaction",
        "instructions": "Choose an appropriate reaction for the avatar based on the user's latest message.",
        "choices": [
          { "value": "neutral" },
          { "value": "happy" },
          { "value": "sad" },
          { "value": "concerned"
        }
      }
    ]
  }
}

Pricing and Availability

• Input Cost: $0.10 per million input tokens (with zero output-token charges).
• Cached Input Cost: $0.01 per million tokens.
• Data Controls: Regional processing is available across the US and EU, alongside Zero Data Retention options for eligible enterprise workflows.

---

Frequently Asked Questions (FAQ)

What is Habitat?

Habitat is OpenAI's proprietary online storage platform built to give products fast, reliable, and multi-tenant access to data, securely handling requests across global regions.

How did OpenAI solve noisy neighbor issues in Azure Cosmos DB?

By working jointly with Azure to implement hot key throttling, Quality of Service (QoS) request tiering, and an improved serverless mode supporting automatic RU-usage-based splits.

What are the main use cases for the Decisions API?

It is optimized for quick, low-cost classification, routing, and scoring tasks in apps and agents, delivering rapid outputs for real-time workflows, voice integrations via GPT-Live, and automated document triage.

How does the Decisions API price requests?

Developers pay solely for input tokens (at $0.10 per million tokens, or $0.01 with prompt caching), with no additional charges for output tokens.