Sometimes it’s best not to trust network vendor datasheets. Nothing quite beats a controlled test of a network device in your lab with your config and your required features. But if you want to load test multiple ports on your 10G device-under-test (or DUT), then things can get very expensive, very fast.
In this post I’ll show a test topology that will help you turn 10Gbps of test traffic into 640Gbps or more.
Test every port at line rate
So before you get excited you should note this is a ‘niche’ test topology. Two or three test ports will cover the vast major of test scenarios. There are very few reasons why you’d need to run line-rate throughput tests on all ports of a DUT at the same time.
Furthermore, this test is hard to pull off on L4-L7 networking devices, as they may lack the ‘internal loopback’ capability needed by the topology. The CPU-processed path of L4-L7 devices are most likely to fail under high throughput conditions. ASIC switched L2 or L3 traffic will most likely transit your DUT unhindered.
Here are some reasons to test using the ‘snake’ topology.
- Burn-in testing. You may want to burn-in a network device before it gets into service. i.e. get it forwarding on all ports at line rate for 24-hours before entering live service.
- Acceptance testing for a customer. I have heard of some customers demanding proof that a device gives the full throughput promised by the datasheet. Probably something a VAR would charge extra for, but hard to execute cost-effectively.
- If full line-rate forwarding is critical to your design. Most of the time it simply isn’t that important. If your design hinges on this assumption, you’d better test it.
- Power draw – If you have a keen electrical engineer, they may want to know the ‘actual’ power draw of your network device when operating at full tilt. The PHY chips on your device can consume a lot of power on a high-density switch, so exercising them will increase your power draw. Over provisioning power can cost your company a lot of money, so this test may deliver the highest return.
Layer 2 Snake test
So that’s the caveats out of the way. How would you test a modern 64 port 10Gbps switch at line rate on all ports at the same time. That’s a whopping 640Gbps of test traffic. In our example we’ll use the Spirent Axon as I know it’s targeted at enterprise customers and is available in a 2 x 10Gbps variant. I’m sure there are other entry-level 10G testers out there which can do a similar job.
The test device connects to the first port on the DUT and then ‘snakes’ the same stream of 10Gbps traffic from the tester through the DUT. The stream of test traffic exists the DUT on it’s last port before returning to the test device.
VLAN Magic
There are some points to note here. You can see that I’ve used vlans to provide the internal loopbacks, so this is a layer-2 switching test. You need n/2 VLANS where ‘n’ is the number of ports you want to test. In the diagram above n = 8, but the main point here is that ‘n’ isn’t limited by the number of test device ports available to you.
By externally bridging the VLANS together, you force the device to switch the same frame ‘n’ times, and exercise all ports at once. Most testers can run a test port bi-directionally so that you can send and receive a duplex 10Gpbs flow.
EDIT: If you’re finding this difficult to understand, remember that these are VLAN access ports. As such the frame that leaves the port from VLAN1 into the loopback patch is ‘untagged’. At this point, the frame doesn’t belong to ‘any’ VLAN. VLAN2 receives the frame from the loopback port and, based on switch configuration for the port, regards that received frame as part of VLAN2… and so on.
You’ll need n x transceivers and n/2 patch cables. NOTE: you are deliberately creating a looped topology with the external patch cables. I just disable STP, but in theory PVST should allow this topology. Just don’t forget to re-set the config after the DUT leaves the lab. Also, please, please isolate your DUT from all networks except serial console before doing any kind of testing.
Layer 3 snake test
You can also test at layer 3 if you want. This time you need to use n/2 VRF instances instead of vlans. You will also need a route in each VRF to reach the test-device’s target IP interface. Note that some devices won’t let you configure this many VRFs, so your mileage may vary. If you’re testing for power consumption or trying to burn-in, then the Layer 2 test will probably suffice.
Sherpa Summary
The snake topology allows you to leverage an entry-level tester for big results. It can help you avoid a big spend if you need to blast a lot of traffic at a modern high-density switch. Regard it as another tool in your testing toolbox.
Disclaimer: Spirent presented the Axon at NFD4 which I attended as a delegate. I have received no incentive for this post. See http://thenetworksherpa.com/disclaimer/ for more details.