Self-Hosted AI Deployment in Atlanta, GA

Payment Processing / Fraud Prevention

An Atlanta payment processor (processing $42B annually, 78,000 merchant accounts, SOC 2 Type II and PCI-DSS Level 1 compliant) was using a third-party fraud detection API that required sending transaction data — including partial card numbers, merchant identifiers, and transaction amounts — to an external provider for scoring. Compliance and operational problems: (1) PCI scope expansion — the third-party fraud API received cardholder data elements (truncated PAN, merchant ID, transaction amount, geographic data) — technically bringing the fraud provider into PCI scope. PCI assessor concerns: the assessor flagging the data transmission as a scope expansion risk and requesting additional documentation of the fraud provider's PCI compliance. The concern: every additional entity in PCI scope increasing compliance complexity and audit burden, (2) latency impact — fraud scoring via external API adding 180-340ms to transaction authorisation — the API call traversing the internet, queuing at the provider, scoring, and returning results. Latency: for a payment processor where authorisation response time directly affected merchant satisfaction and competitive positioning — 200ms of additional latency was significant. Peak period degradation: fraud API latency increasing to 400-600ms during peak transaction periods when the shared API infrastructure was under load, (3) vendor dependency — the fraud provider controlling model updates, scoring thresholds, and API availability. Model changes: the provider periodically updating their fraud models, sometimes changing scoring behaviour in ways that affected the processor's false positive rate. One update: increasing false positive rates by 23% for 48 hours before the provider acknowledged and reverted. The dependency: the processor unable to control the fraud detection behaviour that directly affected merchant experience, (4) cost escalation — the fraud API priced per transaction — $0.008 per scoring request. Annual cost: $3.4M for 425M annual scoring requests. Cost trend: the provider increasing per-transaction pricing 12% annually — a trajectory that would reach $5.4M within 3 years. The escalation: a growing cost for a service where the processor had limited negotiating leverage, and (5) data privacy concerns — merchants and ISOs increasingly asking about data handling for fraud detection. Several large merchants: requiring contractual assurance that their transaction data was not shared with third parties for purposes other than fraud scoring. The fraud API provider's terms: broad data usage rights that made these assurances difficult to provide. The concern: transaction data flowing to a third party that might use it for competitive intelligence or other purposes.

We deployed a self-hosted fraud detection AI system over 12 weeks. Infrastructure deployment (weeks 1-4). GPU infrastructure: NVIDIA A100 GPU cluster deployed within the processor's existing PCI-compliant data centre — 4 GPU servers providing the compute capacity for real-time transaction scoring at the processor's peak volume (8,200 transactions per second). The deployment: AI infrastructure within the PCI boundary, eliminating the scope expansion that external APIs created. High-availability architecture: active-active GPU cluster with automatic failover — ensuring fraud scoring availability matched the 99.99% uptime requirement of the payment processing platform. Load balancing: transaction scoring distributed across GPU servers with automatic rerouting if any server became unavailable. The architecture: self-hosted AI with the reliability that payment processing demanded. Network isolation: the fraud AI infrastructure connected only to the transaction processing network — no internet connectivity, no external API calls. The isolation: complete data containment within the PCI environment. Fraud model development (weeks 2-8). Custom fraud model: ML models trained on the processor's historical transaction data — 3 years of transaction history with known fraud labels. Model architecture: ensemble of gradient boosting (XGBoost) and neural network models — gradient boosting providing interpretable feature-based scoring while neural networks captured complex transaction patterns. Training data: 1.2 billion historical transactions with fraud labels from chargeback and investigation records. Feature engineering: 240 transaction features — velocity features (transaction count and volume in time windows), merchant features (merchant category, geographic, and volume patterns), card features (card type, issuer, usage patterns), and behavioural features (deviation from cardholder spending patterns). The engineering: features designed from the processor's domain expertise rather than generic fraud detection approaches. Model performance: precision and recall tuned for the processor's specific risk tolerance — fraud detection rate of 94.2% with a false positive rate of 0.31% (vs. the third-party provider's 89.8% detection rate with 0.47% false positive rate). The improvement: better detection and fewer false positives because the model was trained on the processor's specific transaction patterns. Real-time scoring deployment (weeks 5-10). Inference optimisation: models optimised for real-time scoring — batch processing of transaction features, model inference, and score generation completing in under 8ms per transaction (vs. 180-340ms with the external API). The optimisation: fraud scoring adding negligible latency to transaction processing rather than the significant delay of external API calls. Scoring pipeline: transaction data flowing from the authorisation system to the fraud scoring service and back — the complete scoring pipeline completing within 12ms end-to-end. The pipeline: real-time fraud scoring that was effectively invisible in the transaction authorisation flow. Score explanation: each fraud score accompanied by the contributing factors — "elevated risk due to: transaction amount 4.2x cardholder average, new merchant category for this card, high-risk geographic region." The explanation: enabling fraud analysts to understand and investigate scored transactions rather than treating scores as black boxes. Model operations (weeks 7-12). Automated model monitoring: model performance tracked continuously — detection rate, false positive rate, and score distribution monitored with automatic alerting when metrics deviated from baselines. The monitoring: detecting model degradation before it affected fraud detection effectiveness. Model retraining pipeline: automated pipeline for periodic model retraining — new transaction data (including recent fraud patterns) incorporated into updated models. Training: occurring on a separate, non-production GPU instance. Model validation: updated models validated against holdout test data before promotion to production. The pipeline: keeping models current with evolving fraud patterns. A/B deployment: new model versions deployed to a fraction of traffic initially — performance compared against the current model before full deployment. The deployment: risk-managed model updates that prevented the surprise behaviour changes that the external provider had caused.

The self-hosted fraud AI deployed across all transaction processing within 14 weeks. Fraud detection performance: fraud detection rate: from 89.8% (third-party) to 94.2% (self-hosted) — 4.4 percentage point improvement. Additional fraud caught: approximately $18.4M in annual fraud losses prevented. False positive rate: from 0.47% to 0.31% — 34% reduction. Recovered legitimate transactions: approximately 680,000 annually — transactions previously declined as suspected fraud now approved. Revenue from recovered transactions: $2.1M annually. Transaction performance: fraud scoring latency: from 180-340ms (external API) to 8ms (self-hosted) — 96% latency reduction. Peak period performance: consistent 8ms scoring regardless of transaction volume (vs. 400-600ms during peaks with external API). Overall authorisation response improvement: 200ms average reduction — measurable merchant satisfaction improvement. Compliance and data control: PCI scope: fraud scoring data transmission to external provider eliminated — PCI scope simplified. PCI assessor: no further findings related to fraud detection data handling. Merchant data assurance: the processor able to contractually assure merchants that transaction data never left its PCI-compliant infrastructure for fraud scoring — resolving the data privacy concerns from large merchant accounts. Model governance: complete control over model updates, scoring thresholds, and behaviour — no more surprise changes from external providers. Cost elimination: annual fraud API cost eliminated: $3.4M/year. Self-hosted infrastructure cost: $480K/year (hardware amortisation + operations). Net annual savings: $2.92M/year. 3-year projected savings: $10.2M (including avoided 12% annual price increases from the external provider). Financial results: additional fraud prevention ($18.4M in caught fraud × processor loss rate): $5.5M/year. Recovered legitimate transactions: $2.1M/year. API cost elimination (net of infrastructure): $2.92M/year. Latency improvement (merchant satisfaction/retention): $1.2M/year estimated. Total annual impact: $11.72M/year. Self-hosted AI deployment: $640,000. Annual operations: $480,000/year. First-year ROI: 1,732%.