P3 - container-runtime/copilot-runtime-bazel-cache-memory-usage-vs-limits-dotcom on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-2-gc-cus-02,kube_container_name:copilot-runtime-bazel-cache,kube_deployment:copilot-r... - 2026-10-16T17:35Z #5170

Alert Details

Source: [P3] [Triggered] container-runtime/copilot-runtime-bazel-cache-memory-usage-vs-limits-dotcom on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-2-gc-cus-02,kube_container_name:copilot-runtime-bazel-cache,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-bazel-cache-production,site:gc-cus-02
Status: CRITICAL
Triggered at: 2026-10-16T17:35Z

This issue was created because the "create_github_issue_on_alert:true" tag was applied to the monitor. This issue was created by ISP.

%%%

Container copilot-runtime-bazel-cache in deployment copilot-runtime-bazel-cache (namespace copilot-runtime-bazel-cache-production, service copilot-runtime-bazel-cache) in cluster dotcom-2-gc-cus-02 — its busiest pod is using 95.955% of this container's memory limit and is at risk of being OOM-killed and restarted.

What this means: the busiest pod's memory usage for this container, relative to the container's own limit. A leak or spike in a single replica will trip this, and crossing the limit triggers an OOM kill.

What to do:

1. Open the Kubernetes dashboard for this container and check per-pod memory — identify the pod climbing toward its limit and whether it is a steady leak or a spike.
2. Investigate top memory consumers / a possible leak; raise this container's memory limit if the workload legitimately needs more.
3. If this is expected, right-size the memory request/limit; otherwise escalate to the service owner.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/memory-usage-vs-limits.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team.

HOW TO OPT OUT

1. Pick ONE of the following exclusions:

   • !catalog_service:<your-service-slug> — excludes all your workloads
   • !kube_deployment:<your-deployment> — excludes one deployment
   • !kube_namespace:<your-namespace> — excludes an entire namespace
   • !kube_container_name:<container> — excludes one container, for example an injected sidecar

2. Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have four filter blocks and the memory monitor has three — the exclusion must appear in every one of them or the filter is inconsistent. The error monitors have one. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

avg(last_15m):max:kubernetes.memory.usage_pct{!catalog_service:github,!catalog_service:token-scanning-service,!catalog_service:pyroscope,!catalog_service:aqueduct,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom} by {kube_deployment,kube_namespace,kube_cluster,site,kube_container_name} * (avg:kubernetes_state.container.memory_limit{!catalog_service:github,!catalog_service:token-scanning-service,!catalog_service:pyroscope,!catalog_service:aqueduct,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name} / avg:kubernetes_state.container.memory_limit{!catalog_service:github,!catalog_service:token-scanning-service,!catalog_service:pyroscope,!catalog_service:aqueduct,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name}) * 100 > 95

The monitor was last triggered at Fri Oct 16 2026 17:35:10 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Tags

alerting_stage:notifying,catalog_service:copilot-runtime-bazel-cache,create_github_issue_on_alert:true,fleet-wide,kube_cluster:dotcom-2-gc-cus-02,kube_cluster_profile:dotcom,kube_container_name:copilot-runtime-bazel-cache,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-bazel-cache-production,managed:true,moda_health_check:kubernetes.memory_usage_vs_limits,monitor,priority:3,priority:p3,routing_mode:fleetwide,site:gc-cus-02,team:compute-platform

On-Call

(none)