Stateful services on AWS: parameter groups, audit logging, and a backup you actually trust

A few years ago I worked on putting a production-grade stateful layer under Pulumi: an RDS Postgres, an ElastiCache Redis, a bastion EC2, and the Ansible glue around them. The provisioning calls themselves were a few dozen lines. The work that mattered was downstream of those — parameter groups, extension management, snapshot rotation, audit logging, the bastion hardening, and a backup story we trusted because we’d actually restored from it.

This is a write-up of what that layer looked like and which decisions earned their keep.

The shape of the project

Three Pulumi stacks: network/ (VPC, subnets, NAT, route tables), core_db/ (RDS Postgres + bastion EC2), core_cache/ (Redis on ElastiCache). The stacks were independent in their lifecycle but referenced each other for shared values:

# core_db/__main__.py
network = pulumi.StackReference("acme/network/prod")
vpc_id = network.get_output("vpc_id")
private_subnet_ids = network.get_output("private_subnet_ids")

StackReference is Pulumi’s mechanism for crossing stack boundaries. We used it deliberately rather than collapsing everything into one stack: the network changed once a quarter at most, the database changed weekly, and mixing those cadences in the same pulumi up was asking for trouble.

Three environments — dev, int, prod — used the same code with different stack config (Pulumi.dev.yaml, Pulumi.int.yaml, Pulumi.prod.yaml). Almost every interesting parameter was a config value, not a literal in code.

RDS Postgres: the parameters that mattered

The instance declaration itself was unremarkable. The parameter group was where the work went:

parameter_group = aws.rds.ParameterGroup(
    f"{name}-pg",
    family="postgres16",
    parameters=[
        # Audit
        {"name": "shared_preload_libraries",
         "value": "pg_stat_statements,pg_cron,pgaudit,pg_tle",
         "apply_method": "pending-reboot"},
        {"name": "pgaudit.log", "value": "all,-read,-function"},
        {"name": "pgaudit.role", "value": "rds_pgaudit"},
        {"name": "pgaudit.log_catalog", "value": "0"},
        # Connection logging
        {"name": "log_connections", "value": "1"},
        {"name": "log_disconnections", "value": "1"},
        # Slow queries
        {"name": "log_min_duration_statement", "value": "1000"},
        {"name": "log_lock_waits", "value": "1"},
        # DDL
        {"name": "log_statement", "value": "ddl"},
    ],
)

instance = aws.rds.Instance(
    name,
    engine="postgres",
    engine_version="16.1",
    auto_minor_version_upgrade=True,
    instance_class=cfg.require("instance_class"),
    allocated_storage=20,
    max_allocated_storage=100,
    multi_az=True,
    backup_retention_period=35,
    deletion_protection=True,
    storage_encrypted=True,
    enabled_cloudwatch_logs_exports=["postgresql", "upgrade"],
    performance_insights_enabled=True,
    parameter_group_name=parameter_group.name,
    skip_final_snapshot=False,
    final_snapshot_identifier=f"snapshot-{name}-final-{int(time.time())}",
    opts=pulumi.ResourceOptions(
        ignore_changes=["password", "engine_version"],
    ),
)

A few of those parameters worth defending:

• pgaudit.log = "all,-read,-function" — log everything except reads and routine function calls. Reads on a busy database produced roughly an order of magnitude more log volume than writes, and logging them all pushed CloudWatch Logs ingest costs into running away. The forensic value of read logs is real but bounded — for most incidents the question is what changed, not what was queried — so we dropped them. Re-enabling reads if compliance ever required it was a parameter-group change.
• max_allocated_storage five times allocated_storage. RDS storage autoscaling is a feature people forget exists. Setting a max meant a disk-fills-up-at-3am scenario didn’t page anyone — RDS handled it. The starting 20 GB looked small for a production database; the working assumption was that real production data would trigger autoscaling within the first week.
• 35-day backup retention. The maximum for automated snapshots, and the right answer when the snapshots were also the point-in-time recovery substrate. Storage is cheap; the expensive part of a backup is having one.
• auto_minor_version_upgrade=True paired with ignore_changes=["engine_version"]. RDS bumped the minor version inside the 16.x track during the maintenance window without a Pulumi diff; major version upgrades were deliberately out-of-band events with their own runbook. The 16.1 in code was a floor, not a freeze.
• final_snapshot_identifier includes a timestamp. Without it, re-running a destroy after a failed destroy failed because the snapshot ID already existed from the first attempt. The timestamp suffix made each attempt unique.
• ignore_changes=["password"]. Passwords were rotated by a separate Lambda; we didn’t want Pulumi fighting with that.
• Multi-AZ on, deletion protection on, performance insights on, storage encrypted on. Multi-AZ doubled cost; the alternative was being down for thirty minutes when an AZ had a bad day. Deletion protection cost nothing.

A read replica existed in production only, with separate parameter group customisations to handle replication delay:

replica = aws.rds.Instance(
    f"{name}-replica",
    replicate_source_db=instance.identifier,
    instance_class=cfg.require("replica_instance_class"),
    parameter_group_name=replica_parameter_group.name,
)

replica_parameter_group = aws.rds.ParameterGroup(
    f"{name}-replica-pg",
    family="postgres16",
    parameters=[
        {"name": "max_standby_streaming_delay", "value": "300000"},  # 5 min in ms
        {"name": "max_standby_archive_delay", "value": "300000"},
    ],
)

max_standby_streaming_delay was the kind of parameter you only learned about the hard way. The default of 30 seconds meant a long analytics query on the replica got killed if the primary wrote during it. Bumping to 5 minutes gave the replica room to breathe; the cost was up to 5 minutes of replica lag during heavy write load. For our workload (analytics, dashboards, BI tooling) that was acceptable. For a workload where the replica handled user reads it wouldn’t have been.

Redis: the parts that needed thinking about

A two-node cluster with multi-AZ and automatic failover, on a small node because the working set was tiny:

replication_group = aws.elasticache.ReplicationGroup(
    name,
    engine="redis",
    engine_version="7.1",
    node_type="cache.t4g.micro",
    num_cache_clusters=2,
    multi_az_enabled=True,
    automatic_failover_enabled=True,
    parameter_group_name="default.redis7",
    auto_minor_version_upgrade=True,
    snapshot_window="02:00-03:00",
    maintenance_window="thu:06:00-thu:07:00",
)

The unobvious bit: automatic_failover_enabled required num_cache_clusters >= 2. The error if you misconfigured was opaque (“Multi-AZ requires at least one read replica”).

Three things this configuration deliberately didn’t do, all of which would have been appropriate for a Redis instance with a more sensitive workload:

• at_rest_encryption_enabled and transit_encryption_enabled were unset — defaulting to off. The cluster lived inside the VPC’s private subnets, the security group accepted traffic only from the application security group, and the cache content was non-PII session/throttle/queue data. Turning encryption on was a one-line change once the use case grew past that envelope.
• snapshot_retention_limit was unset — defaulting to zero, which meant no snapshots were taken regardless of snapshot_window. Redis was treated as a cache; the persistence guarantees came from the source-of-truth Postgres rather than from Redis itself. A workload like payment idempotency keys (where losing the cache would mean double-charging customers during recovery) is what would have forced the retention limit on, alongside encryption.
• cache.t4g.micro is a burstable instance with ~0.5 GB memory. It was the right size for the working set we had and the wrong size for almost any workload that genuinely uses Redis as more than a hint cache. Node type was a config value, so growing into a cache.r7g.large later would have been a stack-config edit.

The bastion: an SSH host listening on 22 and 443

There was one bastion EC2 per environment: a t4g.nano, Ubuntu 22.04 ARM, in a public subnet, with sshd configured to listen on both port 22 and port 443. The 443 listener existed because some developers worked from networks where outbound 22 was firewalled but 443 wasn’t:

bastion = aws.ec2.Instance(
    "bastion",
    ami=ubuntu_arm_22_04,
    instance_type="t4g.nano",
    subnet_id=public_subnet_ids[0],
    vpc_security_group_ids=[bastion_sg.id],
    user_data=cloud_init,
    iam_instance_profile=instance_profile.name,
    key_name=key_pair.name,
)

The cloud-init did the bare minimum sshd hardening in place (no PrintMotd, no X11 forwarding, listen on both ports), then handed off to Ansible for the real configuration. The full hardening pass — fail2ban, auditd, unattended upgrades, monitoring agents, MaxSessions limits — ran as an Ansible playbook against the freshly-provisioned host:

# ansible/roles/bastion_baseline/tasks/main.yml
- name: Limit concurrent SSH sessions per user
  lineinfile:
    path: /etc/ssh/sshd_config
    regexp: "^MaxSessions"
    line: "MaxSessions 3"
  notify: restart sshd

- name: Install and enable fail2ban
  apt:
    name: fail2ban
    state: present
- service:
    name: fail2ban
    enabled: yes
    state: started

- name: Enable unattended-upgrades for security patches only
  apt:
    name: unattended-upgrades
    state: present
- copy:
    src: 50unattended-upgrades
    dest: /etc/apt/apt.conf.d/50unattended-upgrades

We didn’t use SSM Session Manager. The reason was mundane: most database access went through a tunnel (ssh -L 5432:db:5432 bastion), and tunnelling through Session Manager was an order of magnitude more setup-friction than tunnelling through SSH. For straight shell-on-host, SSM is the cleaner answer; for the port-forwarding case it wasn’t.

Postgres-side configuration via Ansible

A surprising amount of database operation lived outside the database creation itself. User and role management, extension installation, scheduled maintenance via pg_cron, monitoring queries — all of these were configuration of the running database, not of the RDS instance. We managed them with Ansible playbooks that connected through the bastion tunnel and executed against Postgres directly:

- name: Install pg_cron extension
  community.postgresql.postgresql_ext:
    db: app
    name: pg_cron

- name: Schedule idle-transaction monitoring
  community.postgresql.postgresql_query:
    db: app
    query: |
      SELECT cron.schedule(
        'log_idle_transactions',
        '* * * * *',
        $$INSERT INTO ops.idle_transactions (sample_at, count)
          SELECT NOW(), COUNT(*)
          FROM pg_stat_activity
          WHERE state = 'idle in transaction'$$
      )
    autocommit: yes

The pg_cron task sampled the idle-transaction count every minute and wrote it to a table; a Grafana panel turned the table into a graph and an alert fired if the count stayed above zero for ten minutes. Idle in transaction is one of those Postgres failure modes that’s invisible until it isn’t — a connection holding locks but doing no work.

The backup story: snapshots were not enough

RDS automated snapshots covered the vast majority of recovery scenarios — restoring a deleted row, recovering from a bad migration, point-in-time recovery within the retention window. They didn’t cover the catastrophic cases: AWS account compromise, a privileged user running a destructive script that touched snapshots too, an entire region gone.

For those, we ran a separate pg_dump to S3, in a different AWS account, every night. The job was one of the cronjobs from the framework I wrote about elsewhere. It tunnelled through the bastion, dumped the database, encrypted the dump with a KMS key in the destination account, and copied it across.

The property that made this worth doing was that the destination account had no admin paths from the source account. Compromising the production account did not give an attacker access to the backups.

The restore drill

Once a quarter, on a Friday afternoon (lower stakes than 3am Tuesday), we restored the latest dump into a fresh RDS instance, pointed a copy of the application at it, and verified a few canary queries. The drill took about ninety minutes end to end.

The first time we ran it we discovered three issues that would have been catastrophic in a real recovery: a missing IAM policy on the destination KMS key, a stale parameter group reference in the restore script, and a pgaudit extension version mismatch that meant the dump wouldn’t import cleanly. Each took five minutes to fix on a Friday afternoon, and would have taken five hours under incident pressure. The drill earned its place in the calendar that day.

What carried forward

A few things, in roughly the order I appreciated them:

1. The parameter group was where the work went — provisioning was nine lines, configuring it correctly was a hundred. Time spent on parameters paid back disproportionately.
2. Multi-AZ, deletion protection, encryption, and storage autoscaling were the floor, not optimisations. Cost-justifying them individually was the wrong frame; they were the cost of running a stateful service in production at all.
3. ignore_changes was the right tool for out-of-band rotations — passwords, engine versions, sometimes parameter group versions. Deciding what was IaC’s responsibility and what wasn’t, and keeping the boundary documented in the code, was as important as the rotations themselves.
4. The bastion was a hardening problem, not a provisioning problem. Cloud-init for the basics, Ansible for the real configuration. A provisioned EC2 was the start of the work, not the end of it.
5. The cross-account dump and the quarterly restore drill were the parts of the backup story that turned an opinion about recoverability into a tested fact. Without the drill, the dump was just storage spend.