I said to a colleague recently, “you can’t get 100% link utilisation on an Ethernet link”. When I tried to explain myself I wished I could link to a simple blog post with a nice graph. So here’s a quick blog post with a nice graph. I have talked a little about link speed in a previous post, but I wanted expand on this and add a quick graph to back up the argument.

Layer 2 Link Utilisation By Packet Size

The layer 2 utilisation of any given Ethernet link will vary by packet size. The reason for this is a fixed gap of 12 bytes inter frame gap plus 7 bytes of start frame delimiter, and a single byte preamble. From a layer 2 perspective we see these gaps as dead space, even though the layer 1 line code is still modulating a signal and transmitting idle or signalling frames.
The net effect is that those 20 bytes of unused bandwidth are fixed and don’t vary by packet size [1]. Those 20 Bytes could mean approximately 24% overhead when you’re using only the minimum frame size of 64 Bytes. This would provide to a maximum possible layer 2 link utilisation of 76%.
There’s no need to panic though. 64 Byte packets are a rarity and don’t represent a typical packet size. However, it’s good to know how your link would respond when faced with these small packet sizes.
The good news is that link utilisation quickly ramps to 96% at more typical packet sizes of 576 Bytes. This wasted link capacity is so small as to be insignificant. Well, I certainly hope you’re not engineering your links to 96%!

What is your average packet size?

I recommend doing a quick check on the average packet size in your network. The calculation is Bytes-per-interval / Packets-per-interval. Below is a sample from a Cisco ‘show interface’ command.
5 minute input rate 442377000 bits/sec, 52505 packets/sec
5 minute output rate 55288000 bits/sec, 43068 packets/sec
Input Average Packet Size = 442377000 / 8(b/B) / 52505 = 1053 Bytes
Output Average Packet Size = 55288000 / 8(b/B) / 43068 = 160 Bytes
The dominant direction is ‘input’ here, and with a 1053 byte average packet size we have nothing to worry about – this is approximately 98% link utilisation. It’s very possible that you’ll have different average packet size in your network.

Sherpa Summary

The quick summary is that very low packet sizes will materially impact your maximum link throughput due to inter-frame overhead, but that this is quick unlikely unless you’re in a lab.
A quick action you can take is to figure out your average packet size for your network.
[1] It is possible to reduce that inter frame gap but this is not commonplace. Also for 10G the inter frame gap is an average of 12 bytes, with the gaps being greater or less than 12 bytes as needed to the full packet plus overhead rounded to 4-byte boundaries. Think MDRR for Inter-frame gaps.