What Changed 

As part of the scheduled weekend maintenance window, the InteleShare infrastructure was switched to a more modern autoscaling and container-scheduling approach using Karpenter. Karpenter selects the most cost-efficient node/instance type for the current cluster load, replacing a prior approach that effectively over-provisioned compute without the team realizing it. The intent was more efficient and cost-effective use of cloud resources, faster scaling in response to traffic, and access to a wider variety of instance types to avoid AWS capacity constraints. 

What Failed 

InteleShare workload has relatively low CPU and memory requirements but handles a very high volume of concurrent network connections. Once the workload moved to Karpenter's "right-sized" scheduling, several instances of that workload's container were placed on smaller nodes than before (moved from a 2XL instance down to an XL, roughly half the CPU). The Linux kernel's connection-tracking table sizes its maximum entries based on the node's CPU and memory. Because the smaller nodes had a much lower ceiling, and because multiple instances of the high-connection workload were scheduled onto the same nodes, the combined connection count exceeded the conntrack limit once Monday-morning production load hit. Once the limit was reached, new connections on the affected nodes experienced delays or timeouts - and because the limit is enforced at the OS level, it affected other containers co-located on the same nodes as well. 

Why It Wasn't Caught Earlier 

The change had already been running in UAT and all other pre-production environments, which the team could observe before rollout, and it behaved correctly there. The issue only manifested under the connection volume and concurrency pattern of full U.S. production traffic - UAT volume is orders of magnitude lower and does not reproduce the same ordering/concurrency of requests. Simulating true production-scale concurrent-connection load in a lower environment is difficult and, at current test capacity, was not something the team could reasonably reproduce ahead of time. 

Impact 

Clients experienced degraded performance - delayed or timed-out connections - for approximately two to two and a half hours on Monday morning (roughly 9:37 a.m. Eastern into the 10-11 a.m. hour), concentrated around peak login/usage volume. 

Resolution 

The on-call platform engineer was paged automatically when the platform-wide Code Blue was triggered. The team identified the affected application tier and reverted that specific component back to the prior scheduling configuration; the rest of the Karpenter rollout (roughly 85%) remains in place. The revert was performed before the underlying root cause (the conntrack ceiling) was fully understood - root cause was confirmed afterward through investigation.