NetworkSherpa

Is CPU or ASIC responsible for forwarding?

John Harrington

I received the question below from reader Ned as a comment on my 24-port ASIC post and thought that the discussion was worth a post of it’s own.

…Would you be able to speak a bit about the actual physical path or packet flow a packet takes inside the switch itself and how does the hardware forwarding take place within the switch and asic. When does packet get sent to the Asic. Is it happen on ingress or on egress? When does packet get analyzed by CPU or control plane. If the CPU never sees the actual packet how does asic know where to forward the packet and does that mean the packets stay within asic itself and is that what is meant to be hardware forwarding. Is Asic = dataplane. Tx

I like this question because it captures a lot of my early assumptions and concerns about data and control plane separation. My response assumes a single-stage modern switch-on-chip ASIC without backplane or fabric. 

Population of the forwarding tables

There is so much going on in a router that it can get very confusing. Thankfully the programming of the forwarding tables is (mostly) done before route-able packets arrive. For example an OSPF LinkStateUpdate (LSU) could announce the availability of a new route. The LSU will arrive into the ASIC at an ingress interface just like all data-plane packets. The destination IP will be that of a router IP or an alias (e.g 224.0.0.5) so the ASIC will forward the entire frame to the CPU for processing. This ASIC-to-CPU interface is normally a PCIe bus. The CPU will run it’s routing process and if accepted, it will program the ASIC forwarding tables with this new route.
When people talk about control-plane and data-plane separation they really mean, “the CPU no longer has to do a prefix lookup for each arriving data-plane packet”. I would call that hardware-based forwarding, rather than control and data plane separation. Note that the control-plane packets still need to transit the ASIC on their way to the CPU.

A data-plane packet arrives

In this context, a data-plane packet is any packet that requires routing to a destination other than the CPU. When these packets arrive at the switch they are placed in one or more packet buffers along with a unique ID. The packet headers and the buffer ID are then copied and processed by the ASIC. The ASIC will consult the forwarding tables and find a match for the prefix we learned from OSPF earlier. These forwarding tables are normally stored in TCAM or Reduced latency-DRAM for ridiculously fast lookup times. Once a next-hop port and IP address have been chosen, the router needs to find the MAC address and consults another table.
Cisco calls this an ‘adjacency table’, and it is populated by the ARP process. ARP entries are aggressively aged out of routers by design, so it’s likely that the required MAC address is not present for the chosen next hop. The ASIC has to go begging to the CPU for help. How degrading.. “What’s that ASIC, you need help? Not so independent now are ya?” (this is how I imagine CPUs talk to ASICS).
Cisco call this an ARP ‘glean’. The CPU will send an ARP request (via the ASIC) and process the response, then program the adjacency table. This works well (until you have millions of ARP gleans), but again it’s not pure hardware forwarding. The CPU is ‘assisting’ the hardware forwarding process in real-time.
Finally we have enough information to construct a new shiny new header and attach it to the previously buffered payload. The temporary buffer information is discarded, and the full frame is queued for transmission out the egress port.

Sherpa Summary

Here’s my quick summary:

I hope it clarifies things rather than further muddying the waters?  Let me know your thoughts in the comments.