DevOps Services in Melbourne, Australia

Financial Services / Fintech

A Melbourne-based fintech (AFSL-licensed, processing A$1.2B in annual transaction volume, 65 engineers across 8 product teams) had a deployment problem. The problems: (1) Deployment velocity: the company deployed to production twice per week — each deployment a coordinated event requiring a "deployment captain," a 2-hour deployment window (typically after 6pm to minimise user impact), and the entire platform engineering team on standby. Rollbacks: manual, taking 30-45 minutes, and requiring the same coordination. The CTO wanted daily deployments with zero-coordination overhead. (2) Infrastructure sprawl: 340 AWS resources created manually through the AWS console over 3 years. Nobody had a complete picture of the infrastructure. Three production outages in the past year traced to: someone deleting an S3 bucket policy they didn't know was production-critical, a security group change that blocked database connectivity, and an EC2 instance type change that reduced memory below the application's requirements. No IaC — the console was the source of truth. (3) Environment inconsistency: developers ran applications locally on macOS, staging ran on a single EC2 instance (versus production's auto-scaling group behind an ALB), and production used services (ElastiCache, MSK) that didn't exist in staging. Bugs discovered in production that couldn't be reproduced in staging: a recurring frustration. (4) Monitoring gaps: CloudWatch metrics existed but nobody looked at them proactively. The alerting strategy: a PagerDuty rule that triggered when the application load balancer returned 5xx errors — meaning the team learned about problems when users were already affected. No distributed tracing, no structured logging, no business-level monitoring (transaction success rates, payment processing latency, API response times).

We rebuilt the deployment infrastructure over 12 weeks: (1) Infrastructure-as-code: imported all 340 AWS resources into Terraform. Organised into modules: networking (VPC, subnets, security groups), compute (ECS Fargate — migrating from EC2), data (RDS PostgreSQL, ElastiCache, MSK), and application services (ALB, CloudFront, S3). Every infrastructure change: submitted as a Terraform PR, peer-reviewed, and applied through the CI/CD pipeline. The AWS console: read-only for production — no manual changes. (2) CI/CD pipeline: GitHub Actions replacing the manual deployment process. Pipeline: PR opened → automated tests (unit, integration, E2E via Playwright) → security scan (Snyk + Semgrep) → build Docker image → deploy to staging automatically → smoke tests against staging → one-click production deployment (merge to main triggers deployment) → automated canary analysis (5% traffic to new version for 10 minutes, automated rollback if error rate exceeds threshold). Rollback: automated — if the canary fails or a manual rollback is triggered, the previous version deploys in under 3 minutes. (3) Environment parity: Terraform workspaces ensuring development, staging, and production use the same architecture — scaled differently (staging: smaller instance sizes, single-AZ) but architecturally identical. Each product team: getting their own ephemeral preview environment deployed from their feature branch — testing against a complete environment rather than a shared staging server. (4) Observability: Datadog across all environments. Structured JSON logging (correlation IDs linking a user request across all microservices), custom metrics (transaction success rate, payment processing latency, API response time by endpoint), distributed tracing (every request traceable from the API gateway through microservices to the database), and SLO-based alerting (alerting on: "transaction success rate dropped below 99.5%" rather than "CPU above 80%"). Business dashboards: real-time visibility into transaction volume, processing latency, error rates, and revenue — the CTO starting each morning with the Datadog dashboard. (5) Security: APRA CPS 234 alignment — the deployment pipeline enforcing: dependency vulnerability scanning (Snyk blocking deployments with critical CVEs), secrets management (AWS Secrets Manager — no secrets in code or environment variables), network security (Terraform-managed security groups — no manual edits), and deployment audit trail (every production change linked to a GitHub PR with reviewer approval).

Deployment velocity: from twice per week (coordinated 2-hour events) to 15+ deployments per day (zero-coordination, automated). Each product team: deploying independently without waiting for other teams or coordinating with platform engineering. The "deployment captain" role: eliminated. Rollback: from 30-45 minutes (manual) to under 3 minutes (automated canary rollback). Outages from infrastructure changes: from 3 per year (manual console changes) to zero (all changes through Terraform PRs with peer review). The "someone deleted something they shouldn't have" category of outages: eliminated entirely. Environment consistency: bugs that "only happen in production" reduced by 80% (the remaining: related to production data volumes, not architectural differences). Preview environments: each PR getting its own complete environment — developers testing against realistic infrastructure before merging. Mean time to detection (MTTD): from "users report it" (minutes to hours) to automated detection within 30 seconds (Datadog SLO monitoring). Mean time to recovery (MTTR): from 45-90 minutes to 8 minutes average (automated rollback + faster diagnosis through distributed tracing). APRA alignment: deployment pipeline documentation satisfying CPS 234 requirements for change management controls. The compliance team: citing the automated deployment audit trail as evidence of robust change management during APRA examinations. Engineering satisfaction: internal survey showing 4.6/5 satisfaction with deployment process (vs. 2.1/5 before). Engineer onboarding: new engineers deploying to production on their first day (vs. 2-3 weeks previously to learn the manual deployment process). Development cost: A$220,000 (12 weeks). Annual value: A$380,000 (engineer time saved from manual deployment and firefighting) + outage cost avoidance (3 outages × A$50,000-200,000 estimated business impact) + engineering velocity improvement (features reaching production 7x faster — competitive advantage in fintech). ROI: development cost recovered within first 7 months from engineering time savings alone.