DevOps Services in San Francisco, USA

AI / Machine Learning / DevOps / San Francisco

A San Francisco-based AI infrastructure company ($82M ARR, providing an AI development platform used by 1,200 enterprise customers to build, train, and deploy ML models, with 380 employees including 180 engineers, Series D funded with $320M raised, and IPO timeline of 12-18 months) needed an MLOps platform transformation to scale operations that were straining under rapid growth. The company's own infrastructure — the infrastructure underlying the product they sold to other companies for ML operations — was ironically managed with practices that were less mature than what their customers expected. Core challenges: (1) GPU cluster management chaos — the company operating 2,800 GPUs (mix of A100 and H100) across 3 cloud providers (AWS, GCP, Azure) for customer workloads. GPU utilisation averaged 34 percent — meaning 66 percent of extremely expensive hardware (approximately $3/hour per GPU) was idle. Annual GPU compute cost: $18M, with $12M estimated waste from low utilisation. Scheduling was primitive — customers reserving GPU capacity in advance and paying whether they used it or not. No preemption, no spot GPU utilisation, and no multi-tenant GPU sharing. The result: large enterprise customers reserving blocks of 64-128 GPUs but typically using 20-30 percent during training runs, while smaller customers waited in queue because "all GPUs were allocated." (2) Model deployment pipeline fragility — the company's customers deploying 4,200 ML models to production monthly through the platform. Model deployment was a 14-step manual process: model packaging, dependency resolution, container building, registry pushing, Kubernetes manifest creation, deployment, health checking, traffic shifting, monitoring setup, A/B test configuration, rollback preparation, documentation, compliance verification, and notification. Each deployment taking 2-4 hours of ML engineer time. Deployment failure rate: 18 percent — primarily from dependency conflicts (model requiring specific CUDA version incompatible with the deployment cluster's GPU drivers), resource misconfiguration (models requesting more GPU memory than available on the target hardware), and configuration errors (incorrect environment variables, missing model weights). Each failed deployment requiring 4-6 hours of debugging. (3) Multi-cloud complexity — the company operating on 3 cloud providers for strategic reasons (customer preference, GPU availability, pricing negotiation leverage). But each cloud had been set up independently — different Kubernetes configurations, different networking, different storage, and different monitoring. An engineer proficient on the AWS deployment could not debug an issue on GCP without significant ramp-up. Terraform code duplicated across clouds with inconsistent variable naming, different module structures, and drift between intended and actual state (34 percent of resources had configuration drift from their Terraform definitions). (4) SOX readiness — with IPO timeline of 12-18 months, the company's auditors (Big Four) identifying significant gaps in deployment controls: no separation of duties (the same engineer could commit code, approve the review, and deploy to production), no documented change management process, no audit trail for infrastructure changes, and production access not restricted (120 of 180 engineers having direct production access). The auditors recommending 12-18 months of remediation — the CFO wanting 6 months. (5) Developer experience — the 180 engineers spending an estimated 30 percent of their time on operational tasks rather than product development. Local development requiring a GPU (not all engineers had GPU workstations), environment setup taking 3 days, and "dependency hell" causing 22 percent of all engineering tickets. New engineer onboarding: 6 weeks before productive contribution. Internal developer satisfaction survey: 2.8/5.0 for "tools and infrastructure" — the lowest score across all categories.

We built an MLOps platform over 10 weeks — unifying GPU management, automating model deployment, and establishing SOX-ready infrastructure governance. (1) GPU orchestration: intelligent scheduling and multi-tenant sharing. Kubernetes GPU scheduling: replacing the reservation-based GPU allocation with intelligent scheduling through custom Kubernetes operators — GPUs allocated dynamically based on actual workload requirements rather than pre-reserved blocks. Multi-tenant GPU sharing: MIG (Multi-Instance GPU) on H100s enabling multiple customer workloads to share a single GPU — smaller inference workloads running on GPU slices rather than consuming entire GPUs. Spot GPU utilisation: non-latency-sensitive workloads (training jobs, batch inference) automatically scheduled on cloud Spot/Preemptible GPU instances — with checkpoint management enabling automatic resume on interruption. Training jobs automatically resuming from the last checkpoint rather than restarting. GPU utilisation monitoring: real-time dashboard showing per-GPU utilisation across all 3 clouds — identifying idle GPUs, underutilised allocations, and scheduling opportunities. Alerts when GPU utilisation dropped below 50 percent for more than 30 minutes. Queue optimisation: intelligent workload queue prioritising jobs based on: customer SLA tier, job deadline, resource efficiency (smaller jobs filling gaps around larger training runs), and cost optimisation (routing to cheapest available cloud with adequate GPU availability). (2) Model deployment automation: 14 manual steps to 1 command. Deployment pipeline: automated model deployment pipeline — ML engineer running `deploy model-v2.3 --target production --strategy canary` and the platform handling all 14 steps automatically. Model packaging: automated dependency resolution and container building — the platform analysing model requirements (framework version, CUDA version, Python dependencies) and building an optimised container image with minimal size. Compatibility checking: before deployment, automated verification that model requirements match target cluster capabilities — CUDA version compatibility, GPU memory requirements, and dependency conflicts checked before any deployment attempt (preventing 85 percent of previous deployment failures). Progressive deployment: canary releases for models — new model version receiving 5 percent of inference traffic, with automated monitoring of: prediction accuracy (compared to previous version), latency, error rate, and business metrics (customer-defined success criteria). Progressive rollout to 100 percent if all metrics meet thresholds, automatic rollback if any metric degrades. A/B testing: built-in model A/B testing — ML teams comparing model versions with statistically significant results, integrated with the deployment pipeline so winning models automatically promoted. Model registry: versioned model registry with complete lineage — every deployed model traceable to: training data version, training configuration, evaluation metrics, and deployment history. Meeting FDA 21 CFR Part 11 requirements for healthcare customers and SOX audit requirements for the company itself. (3) Multi-cloud unification: one platform across three clouds. Unified Terraform: consolidated Terraform codebase with cloud-specific modules — a single variable (cloud_provider = "aws"|"gcp"|"azure") determining which provider modules deployed. Variable naming, resource tagging, and output format standardised across all clouds. Configuration drift eliminated: every resource managed through Terraform with drift detection — weekly automated plans comparing actual state to Terraform state, with drift alerts and automated remediation for non-critical resources. Kubernetes consistency: KubeFed (Kubernetes Federation) managing clusters across all 3 clouds — standardised cluster configuration, network policies, and service mesh (Istio) deployment ensuring consistent behaviour regardless of cloud provider. Cross-cloud workload migration: customer workloads portable between clouds — if GPU availability was constrained on AWS, workloads automatically scheduled on GCP or Azure with equivalent hardware. Customers experiencing no difference in service quality. Unified observability: Datadog as single monitoring platform across all clouds — standardised dashboards, alerting, and APM eliminating the per-cloud monitoring silos. Single pane of glass for: GPU cluster health, model deployment status, customer workload performance, and cost analytics. (4) SOX-ready governance: IPO-grade deployment controls. Separation of duties: pipeline-enforced access controls — developers committing code, reviewers approving (different person), and deployment triggered by the pipeline (not manually by any engineer). Production access restricted from 120 to 12 engineers (SRE team only), with just-in-time access elevation for incident response (automatically revoked after 4 hours). Change management automation: every production change automatically documented — change description (from PR description), risk assessment (automated based on change type and blast radius), test results, security scan results, approval chain, deployment timestamp, and rollback procedure. SOX-compliant change record generated without manual effort. Audit trail: immutable audit log of all infrastructure and deployment changes — every Terraform apply, every kubectl command, every model deployment logged with: who, what, when, from where, and why. Logs stored in a tamper-evident audit account. SOX documentation: automated generation of SOX IT General Controls (ITGC) documentation — deployment process description, access control matrix, change management statistics, and incident response procedures. The CFO's auditors reviewing the automated documentation and confirming "effective controls" within 8 months (versus the 12-18 month timeline initially projected). (5) Developer experience: fast, GPU-accessible, productive. Cloud development environments: Gitpod-based development environments with GPU access — engineers developing and testing ML code in cloud environments with attached GPUs, eliminating the need for GPU workstations and the "it works on my GPU" problem. Environment provisioning: 3 minutes (from 3 days). Standardised tooling: unified development workflow across all engineers — same IDE configuration, same linting rules, same testing framework, same deployment process. New engineer making first deployment within 3 days (from 6 weeks). Self-service: engineers provisioning test clusters, GPU instances, and model serving endpoints through the developer portal without filing tickets. Platform team requests from 60 per week to 12 per week. Documentation: comprehensive developer handbook generated from platform configuration — always current, always accurate, eliminating the "outdated wiki" problem.

Results over 12 months. GPU efficiency: utilisation from 34 percent to 72 percent — a 112 percent improvement. Annual GPU cost from $18M to $11M despite 45 percent growth in customer workloads — $7M annual savings. Spot GPU usage for training workloads saving additional $2.8M. Multi-tenant GPU sharing enabling 3x more concurrent inference workloads without additional hardware. Model deployment: deployment time from 2-4 hours manual to 12 minutes automated. Deployment failure rate from 18 percent to 2.4 percent — compatibility checking preventing most failures before they occurred. Deployments per month from 4,200 (limited by engineer capacity) to 8,400 (automated pipeline removing the bottleneck). Customer model deployment satisfaction improving from 3.2/5.0 to 4.6/5.0. Multi-cloud: configuration drift from 34 percent to 0 percent — every resource matching its Terraform definition. Cross-cloud workload migration enabling 3 customer migrations during cloud provider outages with zero customer downtime. Unified monitoring eliminating the 2-3 hour cloud-specific debugging delay — incidents resolved on any cloud with the same tooling and procedures. SOX: Big Four auditors confirming "effective controls" within 8 months — 4-10 months ahead of the initial 12-18 month projection. IPO timeline maintained. Production access from 120 to 12 engineers without impacting development velocity (just-in-time access for incidents). Compliance team workload: deployment review automated, reducing from 20 hours per week to 3 hours. Developer experience: engineer operational time from 30 percent to 10 percent — recovering approximately 36 engineer-months annually across the 180-person team. New engineer onboarding from 6 weeks to 3 days for first deployment. Developer satisfaction for "tools and infrastructure" from 2.8/5.0 to 4.5/5.0. Engineering attrition from 18 percent to 11 percent — developer experience improvement being the most cited factor. Financial impact: GPU savings $9.8M; deployment automation (customer capacity increase) $4.2M; engineering productivity $6.8M (recovered engineer-months at SF fully loaded cost); SOX acceleration (maintained IPO timeline) — priceless but estimated at $5M+ in valuation preservation; customer satisfaction (reduced churn from deployment quality) $3.8M. Total annual impact: $29.6M (excluding SOX valuation impact). Engagement cost: $620K. Annual platform management: $185K. ROI: 47.7x first year.