OSPF – Setting MTU values for Cisco and Juniper

MTU mismatches are the primary reason an OSPF adjacency becomes stuck in the EXSTART state.  After hellos are exchanged and the routers become neighbors, each OSPF speaker advertises the IP MTU of it’s local interface in a Data Base Description (DBD) LSA.  If there is a mismatch you’ll probably just adjust the configuration to be identical on both ends of the link and be done. However, when you try to peer two OSPF routers with different network operating systems, things start to fall apart fairly quickly.

What do you mean by MTU?

One of the biggest issues I have seen with MTU is that it means different things to different vendors and engineers.  From an OSPF perspective the IP MTU is the maximum size of an IP packet including the headers. OSPF exchanges this information in the initial DBD LSAs so that it can effeciently pack LSAs or LSA headers into IP packets, hoping to avoid IP packet fragmentation or likely loss of the packet.   However every vendor OS seems to have a different way to set the IP MTU, and misconfiguration is the root cause of most OSPF issues.  Let’s explore some examples.

MTU on Cisco IOS

If you’re familiar with Cisco IOS on routers, you’ll know that there are two interface level commands available: ‘mtu’ and ‘ip mtu‘.  Ivan Pepelnjak calls this first mtu the ‘hardware mtu’, and I’ll adopt that term here. All MTU’s are measured in bytes.

The aim of the ‘hardware mtu is to ensure that physical interace can handle the full datagram when layer-2 framing overhead is added to the maximum IP packet size.

The ‘ip mtu’ is the maximum sized IP packet that can be transmitted out an interface. The router will need to fragment an IP packet before transmission if it exceeds the ‘ip mtu’.

On Cisco IOS  the ‘mtu’ and ‘ip mtu’ are set to both set to 1500 bytes by default, which really confused me. How can you have an IP MTU of 1500 bytes and then add an 18-byte ethernet header, but still keep your total frame size under the 1500 byte ‘hardware mtu’?

Well it turns out that the interface controller is programmed with a value of the configured ‘hardware mtu’ + 24 bytes.  (well on my test dynamips 7200VXR anyway)

7200-r1#sh interfaces fa1/0 | i MTU
 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec

7200-r1#sh ip interface f1/0 | i MTU
 MTU is 1500 bytes

7200-r1#show controllers fa1/0 | i mtu
 max_mtu=1524

Note that there are some odd rules for how and when the ‘ip mtu’ and ‘mtu’ will appear in the IOS running config. Best to use the interface commands above and read Ivan’s post to learn more about this. 

It is frustrating that you have to set the hardware MTU indirectly, knowing that IOS will add 24 bytes to the ‘hardware MTU’ you configure.  Life is still simple for ‘ip mtu’,  what you configure (e.g. 1500) will be what OSPF advertises in it’s DBD.  You can verify the IP MTU using the ip-specific show interface command ‘sh ip interface <if_name>’.

NXOS

NXOS has a single MTU command, but it’s meaning is bound to the interface role. For brevity I only consider the Nexus 7000.  The behaviour is different again for Nexus 5000/2000 series switches, which we’ll leave for now.

If you have a layer 2 port (switchport), then you have two choices for the layer-2 frame size. 1500 or 9216 (if system jumbo mtu is enabled).

If you have a default layer 3 port (no switchport), then you can set any value from 576 to 9216 as the advertised ip mtu, still using the ‘mtu’ command.

So for NXOS there is only a single command ‘mtu’.  This keyword signifies ip mtu if it’s a layer-3 interface.  For the layer-2 interface, ‘mtu’ means hardware mtu.  Similar to IOS,  you’re still not configuring the hardware mtu explicitly as there’s a hidden overhead.

IOS-XE

IOS-XE is the OS you’ll see on the lower end ASR routers like the ASR1004.   The MTU configuration on IOS-XE is very similar to IOS. The IOS-XE configuration guide clearly calls out the difference between the configured ‘interface mtu’ and the ‘ip mtu’. This time there’s a 22 byte difference between the configured interface MTU and the actual hardware MTU.  You set the interface mtu using the ‘mtu’ command.  The IOS-XE configuration guide does a good job of breaking out these 22-bytes:

The default interface MTU size accommodates a 1500-byte packet, plus 22 additional bytes to cover  the following additional overhead:

  • – Layer 2 header—14 bytes
  • – Dot1q header—4 bytes
  • – CRC—4 bytes

The IP MTU again is actually pretty straightforward.  You can set it explicitly on the interface directly using an ‘ip mtu’ command.

How big is the ethernet frame overhead?

I know it seems like a softball question.  It’s 18 bytes right?  Well, not so fast.   The ether ‘header’, properly defined as the bytes preceding the ethernet payload is 14 bytes long.   There is also a 4 byte CRC trailer.

Again precision in terminology is important. The ethernet header is 14 bytes, the ethernet trailer is 4 bytes and the total ethernet overhead is 18-bytes.

IOS-XR

This header size issue, becomes really important when you deal with IOS-XR, as seen on ASR9000 and XR-series (formerly GSR) routers.  In IOS-XR there are a few changes in the approach to MTU.

  • You must configure the true hardware MTU explicitly. e.g 1514 for ethernet. There are no hidden overheads added by the router.
  • You do not get to configure the ip ‘mtu’ explicitly, it is derived from the hardware/interface MTU.
  • Ethernet overhead is seen as 14 bytes.
  • XR does not count the CRC trailer when calculating IP mtu.  E.g. a 1514 interface mtu provides a 1500 byte IP MTU.
RP/0/RP0/CPU0:ios#sh ipv4 int gi0/0/0/2 | i MTU
 MTU is 1514 (1500 is available to IP)

I love the fact that you’re in full control of the MTU here, but the 14-byte difference can throw you for a loop the first time you see it. Also not having direct control over the IP MTU is a little odd.

Junos

Junos also takes the ‘you know what you’re doing’ approach to MTU, but provides you full direct access to configure the IP and hardware MTU directly.

  • You can explicitly configure both the hardware and IP MTU. Again, no hidden overheads like IOS/IOS-XE/NXOS.
  • Junos like IOS-XR, does not count the 4-byte CRC trailer as part of the L2 frame overhead.
  • E.g. If the interface mtu is 1514 bytes, junos will not allow you to configure an IP  mtu greater than 1500 bytes.
  • You can examine the hardware mtu and IP mtu directly, using ‘show interface xe-0/0/0 detail | match mtu’
lab@junos-r1> show interfaces xe-0/0/5 detail | match MTU 
 Link-level type: Ethernet, MTU: 1514, Speed: 10Gbps,  <--snip-->
 Protocol inet, MTU: 1500, Generation: 161, Route table: 0

What about Vlan tags?

All the examples above have considered default ethernet frame sizes without VLAN tagging.

  • For IOS-like OSs you get the first vlan tag without needing any additional configuration.  The hidden 22-24 bytes difference between configured interface MTU and actual hardware MTU helps you here.
  • If you want to do Q-in-Q tunnelling, then you’ll need to hack the interface MTU to a higher value e.g. 1504, so that the hardware MTU is sufficiently high to handle the extra tag.
  • For junos and IOS-XE, you configure the overhead explicitly. See below.

MTUs 

Sherpa Summary

OSPF is only concerned with the ‘IP MTU’ of an interface. In general configuring IP MTU is much simpler than configuring the interface or hardware MTU. However it can be hard to determine which MTU is configured on an interface and what value it takes.

  • You cannot assume that two platforms calculate IP MTU in the same way. 
  • You need to clarify exactly what is meant by MTU per platform. Use the configuration guides for your router to
  • Once again, awareness of the potential confusion is the key.
(Visited 70 times, 9 visits today)
Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on RedditBuffer this pageShare on StumbleUpon

12 thoughts on “OSPF – Setting MTU values for Cisco and Juniper

  1. I remember the first time I wanted to bring up ospf between an ex8200 and a nexus in one datacenter and an ex8200 and a 6500 in a different datacenter. It drove me a bit batty that I couldn’t use the same MTU commands, until I figured out what you just discussed, that ip mtu is your friend.

    • Hey Peter,
      Agreed. When it’s available, IP mtu helps to keep things simple. Even when you’re aware of the potential issue, it’s hard to find a good one-page reference. Hoping this post will help me in the future.

  2. Hi John,

    A very nice explanation. I do have a question , which might sound silly probably not related to this topic. So just to keep it simple assuming a packet gets built from application layer down towards layer1. It looks as below
    (l7 data)
    (tcp header =20bytes) — (l7 data)
    (ip header=20bytes) — (tcp header =20bytes) — (l7 data)
    (ethernet header = 14bytes) — (ip header=20bytes) — (tcp header =20bytes) — (l7 data)
    FInally adding the trailer as well to form the ethernet frame
    (ethernet header = 14bytes) — (ip header=20bytes) — (tcp header =20bytes) — (l7 data) — (ethernet trailer = CRC = 4bytes)

    As I understand the size of ethernet || header is 14 bytes (6bytes of src addr + 6 bytes of dst mac + 2 bytes of ether type). I thought even the preamble (7bytes) and start of frame delimiter(1byte) was considered. Guess my assumption was wrong.

    My question is when you said “From an OSPF perspective the IP MTU is the maximum size of an IP packet including the headers”
    => 20bytes ip header + 20bytes tcp header + 1460 bytes l7 data = 1500 right?
    does headers refer to both tcp and ip header ?
    Is the maximum MTU correct assuming data size to be 1460?

    The thing is I kind of understand the concept but difficult to get the wordings.

    -Srinivas

    • Hi Srinivas,
      Thanks for the comment. You’re right when you say that the pre-amble and SFD are not counted as part of the frame.

      The calculations you laid out are solid but the value of 1460 you reach isn’t the IP MTU. 1460 Bytes is the number of bytes available for TCP ‘data’. TCP data is the actual data the upper layer protocols requested TCP to transmit. Another word for this is the maximum segment size or MSS, that is signalled to the TCP peer during the 3-way handshake.

      L2 frame size is 1518 Bytes (18-bytes of over head if you count the 4-byte CRC trailer)
      IP packet size of 1500 Bytes (1480 Bytes available to TCP).
      TCP segment size of 1460 (refers to data section of TCP segement, 20-byte TCP header not counted).

      The weird thing about the TCP MSS size is that it does not include the TCP header, so it’s 1360 1460 Bytes in this example. However the IP MTU size does include the IP header, so it’s 1500 Bytes in this example. When we say an IP MTU of 1500 bytes, the total size of the ip packet including it’s 20-byte header must not exceed 1500 Bytes or it will be fragmented.

      • Awesome that clears my doubt. Thanks John for the clarification. Ya as you said, even I was surprised that TCP MSS size does not include TCP header whereas IP MTU includes IP header as well.

        Also its good that you clearly distinguish TCP data unit as segment, IP data unit as packet and L2 data unit as frame.

        Btw you mentioned “so it’s 1360 Bytes in this example” as TCP MSS , you meant 1460 right?

        – Srinivas

  3. Pingback: » MTU, Junos & Cisco, OSPF Network Happy Time

  4. Pingback: All things Virtual and Storage » Jumbo frames

  5. A couple of clarifications:
    – In IOS, the MTU command is configuring the L2 payload MTU. I.e. IOS adds any necessary extra bytes to account for the L2 encapsulation overhead.
    – IOS XR also supports configuring the L2 payload for IP packets. The command is under ipv4 mtu …You can equivalently set the MTU for IPv6/MPLS packets.

    Thanks,
    Rob

    • Hi Rob,
      Thanks for the comment. I’ll try to add your XR clarification into the XR section.

      Regarding your IOS comment, is L2-Payload-MTU not, by definition, the same as IP MTU? ie. IP MTU always includes the IP header anyway. I understand that IOS adds 24 bytes for framing overhead, but that happens even if I have no Dot1q/ISL configured on the interface. I get the intent, it’s just a bit confusing.

Leave a Reply

Your email address will not be published. Required fields are marked *