Link Utilisation Varies By Packet Size

I said to a colleague recently, “you can’t get 100% link utilisation on an Ethernet link”. When I tried to explain myself I wished I could link to a simple blog post with a nice graph. So here’s a quick blog post with a nice graph. I have talked a little about link speed in a previous post, but I wanted expand on this and add a quick graph to back up the argument.

Layer 2 Link Utilisation By Packet Size

The layer 2 utilisation of any given Ethernet link will vary by packet size. The reason for this is a fixed gap of 12 bytes inter frame gap plus 7 bytes of start frame delimiter, and a single byte preamble. From a layer 2 perspective we see these gaps as dead space, even though the layer 1 line code is still modulating a signal and transmitting idle or signalling frames.
The net effect is that those 20 bytes of unused bandwidth are fixed and don’t vary by packet size [1]. Those 20 Bytes could mean approximately 24% overhead when you’re using only the minimum frame size of 64 Bytes. This would provide to a maximum possible layer 2 link utilisation of 76%.
Link Utilisation by Frame SizeThere’s no need to panic though. 64 Byte packets are a rarity and don’t represent a typical packet size. However, it’s good to know how your link would respond when faced with these small packet sizes.
The good news is that link utilisation quickly ramps to 96% at more typical packet sizes of 576 Bytes. This wasted link capacity is so small as to be insignificant. Well, I certainly hope you’re not engineering your links to 96%!

What is your average packet size?

I recommend doing a quick check on the average packet size in your network. The calculation is Bytes-per-interval / Packets-per-interval. Below is a sample from a Cisco ‘show interface’ command.
5 minute input rate 442377000 bits/sec, 52505 packets/sec
5 minute output rate 55288000 bits/sec, 43068 packets/sec

Input Average Packet Size = 442377000 / 8(b/B) / 52505 = 1053 Bytes
Output Average Packet Size = 55288000 / 8(b/B) / 43068 = 160 Bytes
The dominant direction is ‘input’ here, and with a 1053 byte average packet size we have nothing to worry about – this is approximately 98% link utilisation. It’s very possible that you’ll have different average packet size in your network.

Sherpa Summary

The quick summary is that very low packet sizes will materially impact your maximum link throughput due to inter-frame overhead, but that this is quick unlikely unless you’re in a lab.
A quick action you can take is to figure out your average packet size for your network.
[1] It is possible to reduce that inter frame gap but this is not commonplace. Also for 10G the inter frame gap is an average of 12 bytes, with the gaps being greater or less than 12 bytes as needed to the full packet plus overhead rounded to 4-byte boundaries. Think MDRR for Inter-frame gaps.



, , ,



4 responses to “Link Utilisation Varies By Packet Size”

  1. Shreeram Avatar

    Hi John,
    Nice post. Just a small comment. – In the above post you have mentioned SFD as having 8 bytes. Isnt it that the SFD is actually 1 byte and preemable is 7 bytes ( making the total to 8 bytes ) ?

    1. John Harrington Avatar

      Thanks Shreeram,
      Appreciate the comment, I’ve clarified the post accordingly. There’s a whole pandoras box on the actual codes use to implement this, but best to keep it simple (and accurate ) for now. Thanks again for the feedback,
      /John H.

  2. colin whittaker Avatar

    There is a neat presentation from a nanog a few years ago on sip and hdlc interacting to lose bandwidth
    video of the whole panel session

    1. John Harrington Avatar

      Hey thanks for sharing that Colin, may do a mini post to share it a bit more widely. Hope you’re keeping well!

Leave a Reply

Your email address will not be published. Required fields are marked *