The Token Tipping Point: A CTO’s Guide to LLM Self-Hosting in India

For Indian financial services enterprises, the journey into Generative AI typically starts with an Azure or OpenAI API key. However, as digital-first banking and high-velocity fintech push request volumes toward millions per day, the "rental" model of managed APIs begins to clash with the economic and regulatory reality of "owning" the infrastructure.

In India, this shift is driven by more than just cost; it is about data residency (DPDP Act compliance) and handling the massive concurrency required for a population of 1.4 billion. This guide breaks down the TCO for an Indian enterprise to identify exactly when to flip the switch from API to self-hosted hardware or the emerging "middle path" of local GPU-as-a-Service (GPUaaS).

1. The Architecture: Scaling for India’s Concurrency

An enterprise inference stack for a large Indian bank or NBFC cannot run every task on a massive 70B model. To optimize, we assume a Dual-Model Router architecture:

• The Utility Engine (80% of Volume): A 4B–10B parameter model (e.g., Llama 3 8B) handles low-latency, simple tasks like automated customer support in regional languages or basic document classification.

• The Reasoning Engine (20% of Volume): A 70B+ model (e.g., Llama 70B or Mistral Large) handles complex loan underwriting, fraud detection, and long-context (8k+) analysis.
  

Provisioning for the "India Peak": Unlike global APIs where scaling is the provider's problem, self-hosting requires provisioning for your busiest hour. If your average rate is 1,000 requests/minute, your hardware must handle ~3,000 requests/minute to maintain SLAs during peak transaction hours or festival-season surges.

2. The Economic Breakdown (3-Year TCO in INR)

We compared three strategies: Managed APIs, buying a 4x NVIDIA H100 (80GB) HGX System outright, and using Indian GPUaaS providers.

Cost Category	Managed API (Regional)	Buy Outright (4x H100)	GPUaaS (Reserved)
Initial CapEx	₹0	~₹2.45 Crore	₹0
Monthly OpEx	₹0	~₹1.5–2 Lakh	~₹6.25 Lakh
Monthly API Bill	~₹8 Lakh*	₹0	₹0
Annual Support/Labor	₹0	~₹20 Lakh	₹0
3-Year Total TCO	~₹2.88 Crore	~₹3.8 Crore	~₹2.25 Crore

*Assumes ~10B tokens/month blended volume.

3. The "Inference System" Deep Dive

To achieve performance at scale, your Indian infrastructure requires a sophisticated local software layer:

• Orchestration: Using vLLM or NVIDIA TensorRT-LLM is mandatory. These utilize PagedAttention, allowing GPUs to manage KV Caches efficiently across varying context sizes (500 to 8,000 tokens).

• The GPU Split:

  2x H100s are dedicated to the 70B Model to provide the 140GB+ VRAM required for high-concurrency memory buffers.

  2x H100s run the 8B Model with extreme throughput, ensuring regional language queries return in milliseconds even during traffic spikes.

• Continuous Batching: This ensures that as soon as one token is generated for a user, the GPU immediately starts working on the next request, maximizing your "Sunk Cost" hardware.

  
  

4. When Should You Switch?

In the Indian context, the API remains cheaper until you hit a massive scale because of the 20–30% import duties on high-end silicon. However, the GPUaaS model often beats both options for growing firms.

The "Go" Signal for Moving Off APIs occurs when:

• Volume Crosses 10B Tokens/Month: At this scale, monthly API bills (~₹8.5 Lakh+) exceed the monthly cost of a reserved GPUaaS instance (~₹6.25 Lakh for 4x H100).

• Strict Data Sovereignty: For RBI-regulated entities, keeping PII (Personally Identifiable Information) within a local MeitY-empanelled data centre is often a non-negotiable requirement, making Indian-hosted GPUaaS the default choice.

• Predictable Baselines: If your "trough" (lowest usage) still requires at least 2 GPUs running at 40% utilization, you are wasting money on API margins.
  

The Road Ahead: GPUaaS as the Middle Path

For many Indian enterprises, the leap to CapEx is too steep. A viable middle ground is GPU-as-a-Service (GPUaaS) from Indian providers, which offer H100s at roughly ₹200–500/hour with data residency. This provides the "ownership" benefits of private infrastructure with the "rental" flexibility of APIs, often cutting the break-even point in half and avoiding the headache of tropical cooling costs (₹50k–₹1L/month extra in Indian summers)

NOTE: while this comparison uses "Apple-to-Apple" full-precision models, the math is moving rapidly in favor of self-hosting. New architectural advancements like Mixture-of-Experts (MoE) and Activation-aware Weight Quantization (AWQ) are drastically reducing the VRAM required to achieve "Full Precision" quality. These innovations effectively double your hardware's capacity overnight, cutting the break-even point in half.