Cloud Services in San Francisco

Technology / Artificial Intelligence

A San Francisco-based AI company (Series B, $42M raised, 85 employees — providing an AI-powered document processing API used by 2,800 enterprise customers for intelligent document extraction and classification) needed to re-architect its cloud infrastructure to manage exploding GPU costs, improve inference performance, and build an infrastructure that could support 10x growth without proportional cost increase. (1) GPU cost crisis: inference GPU costs had reached $380K per month — growing at 25% monthly as customer adoption accelerated. At this trajectory, GPU costs would exceed $6M annually within 12 months — consuming the majority of the company's remaining runway. The NVIDIA A100 instances running the inference workload averaged only 35% GPU utilisation — massive waste, but the engineering team lacked the tooling and expertise to optimise utilisation. (2) Inference latency: the document processing API averaged 2.8 seconds per page — acceptable for batch processing but too slow for real-time use cases (which represented 60% of enterprise pipeline). Competitors were achieving sub-1-second latency through model optimisation and serving infrastructure improvements. 5 enterprise prospects ($200K+ ACV each) had cited latency as the primary reason for not proceeding. (3) Training bottleneck: fine-tuning models for enterprise customers (custom extraction models trained on customer-specific documents) required 4-6 hours per model. With 150 active fine-tuning requests per month, the training queue backed up by 3-4 weeks — creating customer onboarding delays and support ticket volume. The training infrastructure (8x A100 nodes) was always at 100% utilisation during business hours and 15% utilisation overnight — poor utilisation efficiency overall. (4) Single cloud risk: all infrastructure ran on AWS us-west-2 — including GPU instances that were frequently unavailable during capacity shortages (the company had experienced 3 incidents where AWS could not fulfil GPU capacity requests during scaling events). Single-provider dependency also weakened negotiating position for enterprise discount agreements. (5) No cost visibility: GPU costs were tracked at the aggregate level but could not be attributed to specific customers, models, or API endpoints. The company could not calculate cost-per-inference (the fundamental unit economics of an AI API), making pricing decisions based on guesswork rather than data.

We re-architected the AI infrastructure over 12 weeks. (1) Inference optimisation: model optimisation — quantisation (FP16 to INT8 for supported models — reducing memory footprint by 50% and increasing throughput by 2.3x with less than 0.5% accuracy degradation), TensorRT compilation (optimising model graphs for NVIDIA inference — 1.8x throughput improvement), and KV-cache optimisation (reducing memory overhead for transformer attention caches). Batching: dynamic batching — accumulating requests and processing them in batches of 4-16 (depending on model and latency constraint), increasing GPU utilisation from 35% to 78%. Routing: model-aware routing — different document types routed to specialised models (smaller, faster models for simple documents; larger models for complex documents), reducing average compute per inference by 40%. Caching: semantic result caching — documents similar to previously processed documents served from cache (embedding-based similarity detection), reducing redundant inference by 25%. (2) GPU fleet management: multi-cloud GPU procurement — inference distributed across AWS (A100, primary), GCP (A100, secondary), and CoreWeave (A100, burst capacity). Spot/preemptible: training workloads on spot instances with automated checkpoint-resume — 65% cost reduction for training compute. Capacity management: automated GPU procurement across providers based on availability and pricing — if AWS GPU capacity is constrained, workloads automatically shifted to GCP or CoreWeave. Reserved instances: 1-year reserved A100 instances for baseline inference demand (covering 60% of steady-state needs) — 40% savings vs. on-demand. (3) Training acceleration: distributed training — fine-tuning jobs distributed across multiple GPU nodes (from single-node to multi-node), reducing training time from 4-6 hours to 45-90 minutes per model. Queue management: priority-based training queue with automated scheduling — enterprise customer fine-tuning jobs prioritised, overnight slots used for non-urgent retraining. Off-peak utilisation: automated scheduling of non-urgent training during overnight hours (midnight-6am) when GPU utilisation was previously 15% — increasing overall utilisation to 72%. Training pipeline automation: end-to-end automation from customer data upload through model training, evaluation, and deployment — reducing manual engineering involvement from 2 hours to 10 minutes per fine-tuning job. (4) Cost attribution: per-inference cost tracking — every API call attributed a compute cost based on GPU time consumed, model used, and instance type. Customer-level unit economics: monthly cost-per-customer calculated, enabling identification of unprofitable customers (high-volume, complex documents on larger models) and pricing optimisation. Model-level economics: cost-per-inference by model type — informing decisions about model selection, quantisation, and routing strategies. Real-time dashboard: GPU utilisation, cost, and throughput visible to engineering and finance teams — enabling informed decisions about capacity, pricing, and optimisation priority. (5) Auto-scaling: inference scaling based on request queue depth and latency percentiles (not CPU utilisation, which is meaningless for GPU workloads). Scale-up: new GPU instances launched within 3 minutes when queue depth indicated increased demand. Scale-down: GPU instances released after 10 minutes of low utilisation — with graceful draining to complete in-flight requests. Predictive: ML-based demand prediction pre-scaling infrastructure before known demand patterns (enterprise batch processing schedules, end-of-month peaks).

GPU costs: from $380K/month to $165K/month (57% reduction) — while handling 85% more inference volume. Projected annual savings: $2.6M. Cost-per-inference: from $0.045 to $0.012 (73% reduction) — transforming the unit economics from marginally profitable to highly profitable at scale. GPU utilisation: inference from 35% to 78%. Training from 55% (business hours only) to 72% (24-hour optimised scheduling). Inference latency: average from 2.8 seconds to 0.6 seconds per page (79% reduction). P99 latency from 8.5 seconds to 1.8 seconds. Real-time use cases: latency improvement unlocked the 60% of enterprise pipeline requiring sub-1-second processing — 5 previously blocked deals ($200K+ ACV each) re-engaged and 3 closed within 6 months. Training velocity: fine-tuning time from 4-6 hours to 45-90 minutes. Training queue from 3-4 weeks to 2-3 days. Customer onboarding time reduced by 65% — new enterprise customers productive within days rather than months. Multi-cloud resilience: zero GPU capacity incidents in 12 months (from 3 the previous year). Automated failover between AWS, GCP, and CoreWeave ensuring continuous availability. Negotiating leverage: multi-cloud credibility enabled 15% better pricing from AWS on enterprise discount agreement. Revenue impact: inference performance improvements and faster onboarding drove 45% ARR growth in the 12 months following re-architecture (from 30% growth trajectory) — approximately $6.3M in additional ARR above baseline trajectory. Annual value: $6.3M (revenue acceleration — faster growth from improved performance and onboarding), $2.6M (GPU cost reduction — 57% reduction in inference and training costs), $1.8M (enterprise deals — 3 previously blocked enterprise deals closed), $0.8M (engineering productivity — automated training pipeline and reduced scaling firefighting) = $11.5M. Development cost: $320,000. Payback: under 11 days.