Skip to content

Send operational alarms to adronhall@proton.me - #64

Merged
Adron merged 1 commit into
mainfrom
hotfix/alarm-email
Oct 22, 2026
Merged

Adron merged 1 commit into
mainfrom
hotfix/alarm-email

Conversation

@Adron

@Adron Adron commented Oct 22, 2026

Copy link
Copy Markdown
Member

Summary

Part of #21. alarm_email was empty, so the operational alarms (5xx rate, Lambda errors and throttles, Aurora near capacity, webhook DLQ, security log) had no recipient. This sets it to adronhall@proton.me, the same address as the billing alerts.

After this deploys, SNS emails a subscription confirmation to that address, and it has to be confirmed before any alarm is delivered.

Test plan

  • terraform fmt -check and make tf-validate
  • The plan adds one aws_sns_topic_subscription
  • Confirm the SNS email after the deploy

🤖 Generated with Claude Code

alarm_email was empty, so the 5xx-rate, Lambda error and throttle,
Aurora capacity, webhook DLQ, and security-log alarms had no recipient.
SNS emails a subscription confirmation after this deploys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

terraform plan (production)

This is exactly what merging will change in AWS.

Plan
random_password.mcp_session: Refreshing state... [id=none]
data.aws_caller_identity.me: Reading...
module.stripe_webhook.aws_cloudwatch_log_group.this: Refreshing state... [id=/aws/lambda/substratal-prod-stripe-webhook]
data.aws_servicequotas_service_quota.lambda_concurrency: Reading...
aws_cloudwatch_log_group.api_access: Refreshing state... [id=/aws/apigateway/substratal-prod]
aws_sns_topic.alarms: Refreshing state... [id=arn:aws:sns:us-east-1:196936048600:substratal-prod-alarms]
aws_sesv2_email_identity.mail: Refreshing state... [id=mail.substratalapps.com]
module.webhook_worker.aws_iam_role.this: Refreshing state... [id=substratal-prod-webhook-worker]
module.api.aws_cloudwatch_log_group.this: Refreshing state... [id=/aws/lambda/substratal-prod-api]
data.aws_route53_zone.main: Reading...
module.mcp.aws_cloudwatch_log_group.this: Refreshing state... [id=/aws/lambda/substratal-prod-mcp]
data.aws_caller_identity.me: Read complete after 0s [id=196936048600]
aws_rds_cluster.main: Refreshing state... [id=substratal-prod-db]
data.aws_servicequotas_service_quota.lambda_concurrency: Read complete after 1s [id=arn:aws:servicequotas:us-east-1::lambda/L-B99A9384]
aws_iam_role.scheduler: Refreshing state... [id=substratal-prod-scheduler]
aws_acm_certificate.api: Refreshing state... [id=arn:aws:acm:us-east-1:196936048600:certificate/a8e2136f-64d3-456b-904a-05ea74eba9b3]
module.jobs.aws_iam_role.this: Refreshing state... [id=substratal-prod-jobs]
module.stripe_webhook.aws_iam_role.this: Refreshing state... [id=substratal-prod-stripe-webhook]
module.mcp.aws_iam_role.this: Refreshing state... [id=substratal-prod-mcp]
aws_apigatewayv2_api.main: Refreshing state... [id=rv9n7e3ww4]
module.api.aws_iam_role.this: Refreshing state... [id=substratal-prod-api]
data.aws_route53_zone.main: Read complete after 1s [id=Z0783627X8ON1GH6I0XB]
aws_secretsmanager_secret.app: Refreshing state... [id=arn:aws:secretsmanager:us-east-1:196936048600:secret:substratal-prod/app-1hXTGg]
module.jobs.aws_cloudwatch_log_group.this: Refreshing state... [id=/aws/lambda/substratal-prod-jobs]
aws_sqs_queue.webhooks_dlq: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/196936048600/substratal-prod-webhooks-dlq]
aws_kms_key.signing: Refreshing state... [id=0d2770cd-bbc8-4904-a432-2e9c9304a3ca]
module.webhook_worker.aws_cloudwatch_log_group.this: Refreshing state... [id=/aws/lambda/substratal-prod-webhook-worker]
aws_kms_key.data: Refreshing state... [id=b0cda5a8-651e-419f-b8ab-61772bce79da]
aws_s3_bucket.audit_archive: Refreshing state... [id=substratal-prod-audit-archive-196936048600]
aws_cloudwatch_metric_alarm.security["refresh-reuse"]: Refreshing state... [id=substratal-prod-security-refresh-reuse]
aws_cloudwatch_metric_alarm.security["destructive-blocked"]: Refreshing state... [id=substratal-prod-security-destructive-blocked]
aws_cloudwatch_metric_alarm.security["failed-logins"]: Refreshing state... [id=substratal-prod-security-failed-logins]
aws_route53_record.dmarc: Refreshing state... [id=Z0783627X8ON1GH6I0XB__dmarc.mail.substratalapps.com_TXT]
aws_route53_record.dkim[2]: Refreshing state... [id=Z0783627X8ON1GH6I0XB_rp4dt5rjjkpnhdyxud6l2spetqo5nom4._domainkey.mail.substratalapps.com_CNAME]
aws_route53_record.dkim[0]: Refreshing state... [id=Z0783627X8ON1GH6I0XB_fmy2g7whqti56v5vn56es6izxo4yzayc._domainkey.mail.substratalapps.com_CNAME]
aws_route53_record.dkim[1]: Refreshing state... [id=Z0783627X8ON1GH6I0XB_fhl63kaowzjfd2splohuyaifnnansx5d._domainkey.mail.substratalapps.com_CNAME]
aws_rds_cluster_instance.main: Refreshing state... [id=substratal-prod-db-1]
aws_cloudwatch_metric_alarm.aurora_near_max: Refreshing state... [id=substratal-prod-aurora-acu-near-max]
aws_apigatewayv2_stage.default: Refreshing state... [id=$default]
aws_cloudwatch_metric_alarm.api_5xx: Refreshing state... [id=substratal-prod-api-5xx]
aws_route53_record.cert_validation["api.substratalapps.com"]: Refreshing state... [id=Z0783627X8ON1GH6I0XB__45a2e898c02bdc2297dde6c5e5a798a8.api.substratalapps.com._CNAME]
terraform_data.app_secret_init: Refreshing state... [id=5474e6c3-4843-79d8-9346-9fc3d3f8fa87]
module.webhook_worker.aws_iam_role_policy.logs: Refreshing state... [id=substratal-prod-webhook-worker:logs]
module.jobs.aws_iam_role_policy.logs: Refreshing state... [id=substratal-prod-jobs:logs]
module.mcp.aws_iam_role_policy.logs: Refreshing state... [id=substratal-prod-mcp:logs]
aws_cloudwatch_log_metric_filter.security["destructive-blocked"]: Refreshing state... [id=substratal-prod-destructive-blocked]
aws_cloudwatch_log_metric_filter.security["failed-logins"]: Refreshing state... [id=substratal-prod-failed-logins]
aws_cloudwatch_log_metric_filter.security["refresh-reuse"]: Refreshing state... [id=substratal-prod-refresh-reuse]
module.api.aws_iam_role_policy.logs: Refreshing state... [id=substratal-prod-api:logs]
module.stripe_webhook.aws_iam_role_policy.logs: Refreshing state... [id=substratal-prod-stripe-webhook:logs]
aws_sqs_queue.webhooks: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/196936048600/substratal-prod-webhooks]
aws_cloudwatch_metric_alarm.webhook_dlq: Refreshing state... [id=substratal-prod-webhooks-dlq]
aws_kms_alias.signing: Refreshing state... [id=alias/substratal-prod-jwt-signing]
aws_kms_alias.data: Refreshing state... [id=alias/substratal-prod-data]
module.mcp.aws_lambda_function.this: Refreshing state... [id=substratal-prod-mcp]
aws_acm_certificate_validation.api: Refreshing state... [id=2026-10-20 20:44:42.323 +0000 UTC]
module.stripe_webhook.aws_iam_role_policy.this[0]: Refreshing state... [id=substratal-prod-stripe-webhook:function]
module.stripe_webhook.aws_lambda_function.this: Refreshing state... [id=substratal-prod-stripe-webhook]
module.webhook_worker.aws_lambda_function.this: Refreshing state... [id=substratal-prod-webhook-worker]
module.webhook_worker.aws_iam_role_policy.this[0]: Refreshing state... [id=substratal-prod-webhook-worker:function]
module.api.aws_lambda_function.this: Refreshing state... [id=substratal-prod-api]
module.api.aws_iam_role_policy.this[0]: Refreshing state... [id=substratal-prod-api:function]
aws_apigatewayv2_domain_name.api: Refreshing state... [id=api.substratalapps.com]
aws_apigatewayv2_api_mapping.api: Refreshing state... [id=zdmjg2]
aws_route53_record.api: Refreshing state... [id=Z0783627X8ON1GH6I0XB_api.substratalapps.com_A]
aws_s3_bucket_public_access_block.audit_archive: Refreshing state... [id=substratal-prod-audit-archive-196936048600]
aws_s3_bucket_server_side_encryption_configuration.audit_archive: Refreshing state... [id=substratal-prod-audit-archive-196936048600]
aws_s3_bucket_versioning.audit_archive: Refreshing state... [id=substratal-prod-audit-archive-196936048600]
module.jobs.aws_iam_role_policy.this[0]: Refreshing state... [id=substratal-prod-jobs:function]
module.jobs.aws_lambda_function.this: Refreshing state... [id=substratal-prod-jobs]
module.stripe_webhook.aws_lambda_alias.live: Refreshing state... [id=arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-stripe-webhook:live]
module.mcp.aws_lambda_alias.live: Refreshing state... [id=arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-mcp:live]
module.api.aws_lambda_alias.live: Refreshing state... [id=arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-api:live]
module.webhook_worker.aws_lambda_alias.live: Refreshing state... [id=arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-webhook-worker:live]
aws_lambda_permission.apigw["mcp"]: Refreshing state... [id=apigateway-mcp]
aws_apigatewayv2_integration.fn["stripe"]: Refreshing state... [id=34l24l9]
aws_apigatewayv2_integration.fn["api"]: Refreshing state... [id=3h3bk4c]
aws_lambda_permission.apigw["api"]: Refreshing state... [id=apigateway-api]
aws_apigatewayv2_integration.fn["mcp"]: Refreshing state... [id=8xk20ia]
aws_lambda_permission.apigw["stripe"]: Refreshing state... [id=apigateway-stripe]
aws_cloudwatch_metric_alarm.lambda_throttles["worker"]: Refreshing state... [id=substratal-prod-worker-throttles]
aws_lambda_event_source_mapping.webhooks: Refreshing state... [id=bed6416f-6a72-474a-83e5-559c995cf8f3]
aws_cloudwatch_metric_alarm.lambda_throttles["api"]: Refreshing state... [id=substratal-prod-api-throttles]
aws_apigatewayv2_route.r["POST /internal/stripe/webhook"]: Refreshing state... [id=bczysio]
aws_apigatewayv2_route.r["$default"]: Refreshing state... [id=cr82zzs]
aws_apigatewayv2_route.r["DELETE /mcp"]: Refreshing state... [id=peme2oj]
aws_apigatewayv2_route.r["GET /mcp"]: Refreshing state... [id=s6ez97n]
aws_apigatewayv2_route.r["GET /.well-known/{proxy+}"]: Refreshing state... [id=kb35wrh]
aws_apigatewayv2_route.r["POST /mcp"]: Refreshing state... [id=a197cl8]
module.jobs.aws_lambda_alias.live: Refreshing state... [id=arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-jobs:live]
aws_iam_role_policy.scheduler: Refreshing state... [id=substratal-prod-scheduler:terraform-339ca2ff2a164d299b046f72e0]
aws_scheduler_schedule.job["audit-archive"]: Refreshing state... [id=default/substratal-prod-audit-archive]
aws_scheduler_schedule.job["webhook-dispatch"]: Refreshing state... [id=default/substratal-prod-webhook-dispatch]
aws_scheduler_schedule.job["stripe-events"]: Refreshing state... [id=default/substratal-prod-stripe-events]
aws_scheduler_schedule.job["settings-projections"]: Refreshing state... [id=default/substratal-prod-settings-projections]
aws_scheduler_schedule.job["entitlement-sweep"]: Refreshing state... [id=default/substratal-prod-entitlement-sweep]
aws_scheduler_schedule.job["subscription-access"]: Refreshing state... [id=default/substratal-prod-subscription-access]
aws_scheduler_schedule.job["erasure-cascade"]: Refreshing state... [id=default/substratal-prod-erasure-cascade]
aws_cloudwatch_metric_alarm.lambda_errors["worker"]: Refreshing state... [id=substratal-prod-worker-errors]
aws_scheduler_schedule.job["push-maintenance"]: Refreshing state... [id=default/substratal-prod-push-maintenance]
aws_scheduler_schedule.job["seat-sync"]: Refreshing state... [id=default/substratal-prod-seat-sync]
aws_cloudwatch_metric_alarm.lambda_errors["api"]: Refreshing state... [id=substratal-prod-api-errors]
aws_cloudwatch_metric_alarm.lambda_errors["jobs"]: Refreshing state... [id=substratal-prod-jobs-errors]
aws_cloudwatch_metric_alarm.lambda_errors["mcp"]: Refreshing state... [id=substratal-prod-mcp-errors]
aws_cloudwatch_metric_alarm.lambda_errors["stripe"]: Refreshing state... [id=substratal-prod-stripe-errors]
aws_scheduler_schedule.job["webhook-health"]: Refreshing state... [id=default/substratal-prod-webhook-health]
aws_scheduler_schedule.job["cleanup"]: Refreshing state... [id=default/substratal-prod-cleanup]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # aws_sns_topic_subscription.alarms[0] will be created
  + resource "aws_sns_topic_subscription" "alarms" {
      + arn                             = (known after apply)
      + confirmation_timeout_in_minutes = 1
      + confirmation_was_authenticated  = (known after apply)
      + endpoint                        = "adronhall@proton.me"
      + endpoint_auto_confirms          = false
      + filter_policy_scope             = (known after apply)
      + id                              = (known after apply)
      + owner_id                        = (known after apply)
      + pending_confirmation            = (known after apply)
      + protocol                        = "email"
      + raw_message_delivery            = false
      + region                          = "us-east-1"
      + topic_arn                       = "arn:aws:sns:us-east-1:196936048600:substratal-prod-alarms"
    }

  # module.api.aws_lambda_alias.live will be updated in-place
  ~ resource "aws_lambda_alias" "live" {
      ~ function_version = "7" -> (known after apply)
        id               = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-api:live"
        name             = "live"
        # (5 unchanged attributes hidden)
    }

  # module.api.aws_lambda_function.this will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "substratal-prod-api"
      ~ last_modified                  = "2026-10-22T03:40:45.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-api:7" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-api:7/invocations" -> (known after apply)
      ~ s3_key                         = "364ce5bb0458b14bd90d076a972d99d7e3fecfcf/api.zip" -> "3c8a41ff003e8dc7ab1f7d76719cf90510a42b1d/api.zip"
        tags                           = {}
      ~ version                        = "7" -> (known after apply)
        # (28 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.jobs.aws_lambda_alias.live will be updated in-place
  ~ resource "aws_lambda_alias" "live" {
      ~ function_version = "7" -> (known after apply)
        id               = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-jobs:live"
        name             = "live"
        # (5 unchanged attributes hidden)
    }

  # module.jobs.aws_lambda_function.this will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "substratal-prod-jobs"
      ~ last_modified                  = "2026-10-22T03:40:45.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-jobs:7" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-jobs:7/invocations" -> (known after apply)
      ~ s3_key                         = "364ce5bb0458b14bd90d076a972d99d7e3fecfcf/jobs.zip" -> "3c8a41ff003e8dc7ab1f7d76719cf90510a42b1d/jobs.zip"
        tags                           = {}
      ~ version                        = "7" -> (known after apply)
        # (28 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.mcp.aws_lambda_alias.live will be updated in-place
  ~ resource "aws_lambda_alias" "live" {
      ~ function_version = "8" -> (known after apply)
        id               = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-mcp:live"
        name             = "live"
        # (5 unchanged attributes hidden)
    }

  # module.mcp.aws_lambda_function.this will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "substratal-prod-mcp"
      ~ last_modified                  = "2026-10-22T03:40:45.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-mcp:8" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-mcp:8/invocations" -> (known after apply)
      ~ s3_key                         = "364ce5bb0458b14bd90d076a972d99d7e3fecfcf/mcp.zip" -> "3c8a41ff003e8dc7ab1f7d76719cf90510a42b1d/mcp.zip"
        tags                           = {}
      ~ version                        = "8" -> (known after apply)
        # (28 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.stripe_webhook.aws_lambda_alias.live will be updated in-place
  ~ resource "aws_lambda_alias" "live" {
      ~ function_version = "7" -> (known after apply)
        id               = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-stripe-webhook:live"
        name             = "live"
        # (5 unchanged attributes hidden)
    }

  # module.stripe_webhook.aws_lambda_function.this will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "substratal-prod-stripe-webhook"
      ~ last_modified                  = "2026-10-22T03:40:45.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-stripe-webhook:7" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-stripe-webhook:7/invocations" -> (known after apply)
      ~ s3_key                         = "364ce5bb0458b14bd90d076a972d99d7e3fecfcf/stripe-webhook.zip" -> "3c8a41ff003e8dc7ab1f7d76719cf90510a42b1d/stripe-webhook.zip"
        tags                           = {}
      ~ version                        = "7" -> (known after apply)
        # (28 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.webhook_worker.aws_lambda_alias.live will be updated in-place
  ~ resource "aws_lambda_alias" "live" {
      ~ function_version = "7" -> (known after apply)
        id               = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-webhook-worker:live"
        name             = "live"
        # (5 unchanged attributes hidden)
    }

  # module.webhook_worker.aws_lambda_function.this will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "substratal-prod-webhook-worker"
      ~ last_modified                  = "2026-10-22T03:40:45.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-webhook-worker:7" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:196936048600:function:substratal-prod-webhook-worker:7/invocations" -> (known after apply)
      ~ s3_key                         = "364ce5bb0458b14bd90d076a972d99d7e3fecfcf/webhook-worker.zip" -> "3c8a41ff003e8dc7ab1f7d76719cf90510a42b1d/webhook-worker.zip"
        tags                           = {}
      ~ version                        = "7" -> (known after apply)
        # (28 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

Plan: 1 to add, 10 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

@Adron
Adron merged commit 331dd46 into main Oct 22, 2026
8 checks passed
@Adron
Adron deleted the hotfix/alarm-email branch October 22, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant