01·THE PROBLEM
No Terraform, no Terragrunt, no infrastructure as code of any kind. Every resource had been clicked into existence in the console, while the platform grew to 250,000 users a month. Releases went out roughly twice a day and each one needed the person who remembered how it was wired.
02·WHAT WE BUILT
Three isolated AWS accounts generated from one Terragrunt codebase: a three tier VPC, three purpose built ECS clusters behind a service mesh, tuned MySQL with rotating credentials, and an identical CodePipeline for all 49 repositories.
03·THE RESULT
More than ten times the production deploy rate, on zero downtime rolling releases behind health checks and a circuit breaker. A developer joins a team, gets their repos, and ships on day one without learning the platform first.
INSIDE THE PLATFORM
The decisions underneath the platform.
→
Three accounts, one codebase. Production, development and a shared root account, each with its own state bucket, lock table and assume role. Development runs on a completely separate domain, so a stray host header in dev cannot reach a production customer.
→
A VPC in three tiers. Three availability zones, each with a public, private and database subnet. Public carries IPv6 and the load balancer, applications sit in private behind NAT, and the database tier is created with no NAT route and no internet gateway route at all. Flow logs archive to S3.
→
Three ECS clusters, on purpose. Backend services, the public API gateway and the brand site fleet each get their own cluster, so a spike on the marketing sites cannot starve booking or payments. Capacity is Graviton on spot with a bid ceiling, autoscaling four to eight instances, Container Insights on.
→
MySQL tuned, not just provisioned. skip_name_resolve drops a reverse DNS lookup from every connection, innodb_read_io_threads is raised to 16 for the read heavy workload, and long_query_time at 2 seconds feeds slow query logs into CloudWatch alongside Performance Insights.
→
Data split, not pooled. Multiple databases and multiple EFS filesystems across regions, so no single filesystem or database instance is a shared point of failure. Resilience is what let deploys become frequent and boring.
→
Credentials nobody knows. Secrets Manager rotates database credentials on a schedule from a Lambda inside the isolated subnets. A CloudWatch rule catches the rotation succeeded event and rolls the affected ECS services, so applications pick up the new secret with nobody paged.
→
One pipeline pattern, 49 repositories. Every service and every brand site gets the same generated chain: CodeStar connection to GitHub, CodeBuild to ECR, then an ECS deploy. A new repository inherits the whole thing instead of being wired by hand.
→
Deploys stopped being events. Releases roll out behind load balancer health checks with the deployment circuit breaker on, so a bad task never takes traffic and a failed rollout reverses itself. Deploy frequency rose more than tenfold with no maintenance window on any platform.
→
Onboarding is a team membership. The GitHub organization, its teams and their repository permissions are all managed as code. A developer joins a team, gets exactly the repos that team owns, and pushes to a pipeline that already exists.
THE LOAD IT CARRIES
| Users a month | 250,000 | Across the marketplace and its brands |
| API calls a month | 50,000,000+ | Through the gateway into the mesh |
| Per user, per month | ~200 calls | Roughly 50 on a single session |
| Average throughput | ~19 req/s | Sustained across the month |
| Peak throughput | ~100 to 200 req/s | Five to ten times average |
| Origin response time | ~25 ms average | Measured at the gateway, network to the visitor excluded |
| Compute carrying it | 8 to 16 vCPU | Four to eight Graviton spot instances |
| Backend services | 25 | Behind App Mesh and Cloud Map |
| Brand sites | 23 | Each with its own domain and mail identity |
We cannot name the client: the engagement is covered by an NDA and the marketplace is described here in anonymized form. The figures above are published with the client's explicit permission.
THE BUILD IN DETAIL
The platform now lives in one repository that describes three AWS accounts across two regions. Everything below is defined there, versioned, and reviewable.
Three accounts, one codebase
- Production, development and a shared root account, each with its own Terraform state bucket, DynamoDB lock table and assume role, so a credential for one account cannot touch another
- Development runs on a completely separate domain from production, so a misrouted host header or a stray DNS record in dev can never reach a real customer
- Both environments are generated from the same modules with different inputs, which is what makes a change that worked in development a fact rather than a hope in production
How the VPC is structured
- Three availability zones, each carrying a public, a private and a database subnet, so losing a zone costs capacity rather than the platform
- Public subnets carry IPv6 and the internet gateway and hold only the load balancer and edge. Every application runs in the private tier behind NAT
- The database tier is created with no NAT route and no internet gateway route at all, so the data subnets are unreachable from the internet by construction rather than by firewall policy
- VPC flow logs archive to S3 for traffic forensics, and interface endpoints keep ECS and ECR traffic on the AWS network instead of routing it out over the internet
How ECS is structured
- Three separate clusters, one for the backend services, one for the public API gateway and one for the brand site fleet, so a traffic spike on the marketing sites cannot starve the booking or payment path
- Graviton EC2 capacity on spot instances with a bid ceiling, autoscaling from four to eight instances, with Container Insights on for task level visibility
- A second capacity provider dedicated to background work, so queue workers and scheduled jobs never compete with request traffic on the same instances
- App Mesh with Envoy sidecars for service to service traffic and Cloud Map for discovery, so services resolve each other by name rather than by hardcoded address, with one shared file assigning every service its port
- A self hosted nginx API gateway fronting the mesh, which replaced the legacy API Gateway stages and could fall back to them during the migration
Data tier: tuned and split for resilience
- Storage and data are split rather than pooled: multiple databases and multiple EFS filesystems across regions, so no single filesystem or database instance can take the platform down with it
- MySQL 8 on Graviton RDS with storage autoscaling, so growth does not need a maintenance window
- skip_name_resolve enabled to drop a reverse DNS lookup from every single connection, which matters when 25 services all hold pools
- innodb_read_io_threads raised to 16 for the read heavy marketplace workload, utf8mb4 enforced end to end, and innodb_file_per_table set deliberately rather than left at the default
- long_query_time set to 2 seconds with slow query, audit and error logs exported to CloudWatch, and Performance Insights on, so the next thing to optimize is measured rather than guessed
- Encryption at rest, IAM database authentication, deletion protection, 30 day backup retention with a final snapshot, and a maintenance window outside business hours
- Credentials rotate on a schedule through Secrets Manager using a rotation Lambda that runs inside the isolated database subnets. A CloudWatch rule listens for the rotation succeeded event and triggers a second Lambda that rolls the affected ECS services, so applications pick up the new credential with nobody paged
Pipelines across 49 repositories
- Every backend service and every brand site gets the same generated pipeline: a CodeStar connection to GitHub, CodeBuild to build and push to ECR, then an ECS deploy. A new repository inherits the whole chain instead of being wired by hand
- Releases roll out behind load balancer health checks with the deployment circuit breaker enabled, so a bad task never takes traffic and a failed rollout reverses itself. No maintenance windows on any platform
- The brand fleet builds its shared theme and nginx image once in a parent pipeline, then a Lambda fans the execution out across all 23 child pipelines, so a change to shared code reaches every site in one run
- The GitHub organization is managed as code, including teams and the repository permissions attached to them, so onboarding a developer is a team membership rather than a checklist
The brand fleet
- 23 white label sites, each its own ECS service with its own domain, certificate, CloudFront distribution and host header routing rule
- A per brand SES identity with IAM narrowed to that brand, so one site can only ever send mail as itself
- A CloudFront invalidation Lambda wired into deploys, so a release is live at the edge without a manual cache purge
Workflow and visibility
- Atlantis GitOps: plan on every pull request, apply gated behind approval, parallel plan and apply across modules, source branches cleaned up on merge
- Application performance monitoring through Dynatrace and centralized logging through a Kinesis Firehose stream into Sumo Logic, enabled per service from the same shared config
- Remote state in per account S3 buckets with DynamoDB locking, so two engineers can never apply over each other
The AWS services this platform connects
VPC and VPC endpoints, EC2 with Auto Scaling, ECS, ECR, App Mesh and Cloud Map for the compute and mesh layer. RDS, ElastiCache, EFS, S3 and DynamoDB for state. CloudFront, ACM, Route 53 and Application Load Balancer at the edge. Lambda, SQS, SNS, SES and API Gateway for eventing and messaging. KMS, Secrets Manager, SSM Parameter Store and IAM for identity and encryption. CodePipeline, CodeBuild and CodeStar Connections for delivery. CloudWatch, CloudTrail and Kinesis Data Firehose for logs, events and audit.
Want us to build yours?
This is the kind of product we design, build and ship end to end: fixed scope, production-grade, live in weeks. Tell us what you're building and we'll say straight if it's a fit.