Why does my Amazon EC2 instance exceed its network limits when average utilization is low?

Key Takeaways

[music] Hello, I'm Rakkesh, a cloud support engineer here at the AWS office in Bangalore. Today I'm going to explain why your Amazon EC2 instance may exceed its network limits even when the average utilization appears low. Let's get started. When you query network performance metrics in real time on instances that support enhanced networking through the elastic network adapter or ENA, you might notice the following metrics. Bandwidth inbound allowance exceeded. The number of packets ceued or dropped due to inbound bandwidth exceeding the maximum. Bandwidth outbound allowance exceeded. The number of packets skewed or dropped due to the outbound bandwidth exceeding the maximum. Packets per second allowance exceeded. The number of packets skewed or dropped due to the packets per second or PPS exceeding the maximum. Connection tracking allowance exceeded the number of packets skewed or dropped due to connections exceeding the maximum trackable. Even if the average bandwidth or PPS shown in the Amazon Cloudatch appears low, you might still see packets being ceued or dropped. The most common reason for this is micro bursts. Microws are short spikes in network demand followed by periods of low activity. Towardatch metrics are sampled every 60 seconds and aggregated over 5 minute periods. So micro burst lasting just seconds or milliseconds might not be fully captured. Let's look at an example. Your instance has 10 Gbps network bandwidth limit. In one 60-cond sample, you transfer 20 GB of outbound data and use up all available bandwidth for around 20 seconds before going idle. While the 5 minute average is low at around.5 Gbps, you still reach the 10 Gbps limit during that micro burst. To properly monitor micro bus, use operating system tools to monitor network statistics more granularly, such as S, network load, or interface top on Linux or performance monitor on Windows. The Cloudatch agent can also publish custom metrics at 1 second resolution. However, this option incurs higher charges. It's also best practice to monitor the ENA metrics directly. The driver version must be 2.2.10 or higher on Linux or 2.2.2 or higher on Windows. Cloudatch agent can publish these ENA metrics as well. There are two ways to monitor ENA metrics. Turn on detailed monitoring. Publish ENA metrics to Cloudatch. To turn on detailed monitoring, follow these steps. Open the Amazon EC2 console. In the navigation pane, choose instances, select the instance, choose actions, select monitor and troubleshoot, and then choose manage detailed monitoring. On the detailed monitoring page, do one of the following. For detailed monitoring, select enable. For basic monitoring, clear enable. Choose confirm. Now, let's publish ENA metrics to Cloudatch. Create new IM role with Amazon SSM managed instance core and Cloudatch agent server policy attached. Attach the newly created IM role to the instance. Log to the EC2 Windows instance. Create a config file config.json at the location C program files Amazon Amazon Cloudatch agent. Restart the Cloudatch agent with the new configuration. To do this, run the a fetch config command in PowerShell as an administrator. To view network performance metrics in the Cloudatch console, follow these steps. Open the Cloudatch console. In the navigation pane, choose metrics. Choose the name space for the metrics collected by the agent. By default, this is Cloudatch agent. Choose a metric dimension such as per instance metrics. To prevent microw traffic at the sender side to avoid exceeding throughput or packet rate limits. This requires application changes and possible OS support. If limits are frequently reached, then consider scaling up to a larger instance size with higher network limits or scaling out across multiple instances. For Linux, some additional mitigations include the following. Using the socket option maximum pacing rate to pace individual sockets. Using queuing disciplines such as fade queueing to smooth traffic bursts. Reducing the kernel transmission Q length from 1024 packets default. Thanks for watching and happy cloud computing from all of us here at AWS. [music]