A friend of mine has just ordered a shiny new packet generator for his network lab. I’ve spent some time working as a QA engineer in a network lab and wanted to share some advice.
You can purchase stateful and stateless packet generators from major vendors like Spirent, IXIA or Agilent. If you just need to test throughput, latency or loss, a stateless packet generator will do the trick. The test hardware will use an ASIC to produce line-rate 10G traffic or higher. The Cisco Enterprise Testing Book calls this a ‘bit-blaster’ which I love. In the wrong hands it can also be a ‘network-melter’.
You need a stateful packet generator if you want to test your routing protocols in conjunction with traffic load. A stateful packet generator such as Ixia’s IxNetwork, will use dedicated CPUs to form and maintain adjacencies, inject routing protocol packets, etc. You can use the stateful feature to inject prefixes which are then used as test targets by high-rate stateless traffic.
Licensing is a major source of pain when operating a stateful packet generators. There are often licenses required per protocol and even per-combination of protocols. For example, I had to buy a license for LACP and OSPF individually but there was also an additional license to use OSPF over a LACP LAG. You should thoroughly investigate feature licensing before you choose your vendor or cut that first purchase order.
There is a steep learning curve for packet generators, mainly because they are GUI configured and windows-based. I’m sure the GUI was user friendly at the start, but with the explosion in features and options the UX is pretty poor. These systems are now crying out for a structured CLI.
The packet-generators also behave differently from regular routers, in that they will blast packets out a TX interface regardless of path health and whether they have a route to the target IP address or not. This allows the packet generator to measure loss and convergence, where the loss and re-appearance of test frames is recorded by the RX port of the generator.
A big risk with load testing is that a 10Gig flow could escape the lab into your enterprise network. This can easily happen if you’re injecting a line rate traffic flow into a device-under-test (DUT) and you lose your route to the flow destination. The test traffic flow will often match the default route and leak into the lab-management network, potentially taking your enterprise network off the air. Not that I have ever, (repeatedly) taken a lab off air mind you! You can mitigate this by disconnecting the DUT mgmt ethernet interface, place it in a mgmt VRF or use more specific routes back to your enterprise network.
There is a step learning curve for these tools and you can easily lose hours of your time with nothing to show for it. I recommend the following steps sequentially to gain the most productive use of your time.
- Start small – A single DUT with two ports connected to packet generator
- You don’t have CDP. Toggle interfaces on DUT and tester to ensure you’re on the right port.
- Ensure you have ARP resolution before you start your traffic flows.
- Test a single flow at a very low traffic rate. Verify you’re receiving no packet loss before continuing.
- You can then ramp up the line rate to test for errors, throughput etc.
- Or.. you can keep the line rate low and add dynamic routing for a single DUT, adding the prefixes as flow targets.
- When you’re happy with the performance of a single DUT, you can move on to a system-under-test (SUT).