Using Dual 10 Gigabit Ethernet on the USRP X300/X310

Application Note Number

AN-296

Abstract

This short guide is meant to help in quickly setting up an X-series USRP for use over two 10 Gigabit Ethernet links simultaneously.

Overview

Disclaimer: In order to run at full receive rate (200 Msps) on two channels, it is recommended to have a multicore machine with clock frequencies above 3 GHz per core.

Ettus recommends using the Intel Ethernet Converged Network Adapter X520-DA2 for X-series USRPs over ethernet. This guide also assumes the user has the required software (Xilinx tools such as Vivado) to build the X-series FPGA image.

Build Steps

Grab the latest UHD master by running the following commands:

   $ git clone https://github.com/EttusResearch/uhd.git
   $ cd uhd/host && mkdir build
   $ cmake ../
   $ make test
   $ make install

Next, build the latest XG image:

   $ git clone https://github.com/EttusResearch/fpga.git
   $ cd fpga/usrp3/top/x300
   $ make X300_XG 

OR

   $ make X310_XG

This puts an image in build-X3XX_XG.

Device Setup

Ensure that the device is available for communication by pinging its initial address (usually 192.168.10.2 on the HG/S image). In this step, the XG image will be burned into memory for the X300 or X310 FPGA:

   $ uhd_image_loader --args="type=x300,addr=<IP ADDR>" --fpga-path="<path to the image we just built with .bit extension>"

• Note: make sure your image matches your X-series USRP, as the sizes are different between the X300 and X310.*

We should always have the MTU size above 8000 (make it 9000) for both 10GigE interfaces. This can usually be done with ifconfig:

   $ sudo ifconfig <interface eth0, eth1, etc.> mtu 9000

Make sure you can find both USRP links with dual 10GE and ping the device on both IPs (try 192.168.30.2 and 192.168.40.2 by default).

Issues can arise with the NIC buffering, so it is a good idea to set the maximum number of descriptors with ethtool.

In order to get the maximum number of descriptors, run the following:

   $ sudo ethtool -g <interface>

To set the number of descriptors:

   $ sudo ethtool -G <interface eth0, eth1, etc.> rx <max RX from previous command>

To test the installation and USRP, try running benchmark_rate, which can be done using the whole path to the command or from the directory in which it is installed; we use the latter here.

   $ ./benchmark_rate --args="type=x300,addr=<First IP, typically 192.168.30.2>,second_addr=<Second IP, typically 192.168.40.2>" --channels 0,1 --rx_rate 200e6 --duration 30

Note: the order of the provided IPs should not matter for testing. The front panel LEDs can be used for verifying IP/channel mapping.

Note: When using UHD 3.15 or prior, only the <First IP> link will be used by default for TX only; both links are used for RX by default. If the aggregate TX data rate is more than can be provided by one link, then there are special flags to add to the "--args" section to support dual-link data transport ('load balancing' of a sort): "enable_tx_dual_eth=1,skip_dram=1". If the aggregate TX data rate is less than that provided by one link, then the primary link will be used regardless of these special flags. A commandline using these special flags could be:

   $ ./benchmark_rate --args="type=x300,addr=<First IP, typically 192.168.30.2>,second_addr=<Second IP, typically 192.168.40.2>,enable_tx_dual_eth=1,skip_dram=1" --channels 0,1 --rx_rate 200e6 --duration 30

Host Tuning

If overflows are encountered (or drops if any buffers are overwhelmed), the CPU might be throttling. This can be checked with cpufreq or cpupower.

As an example, consider running cpufreq:

   $ cpufreq-info

This command tells us the current state and min/max frequency of the CPU cores.

To set the maximum, run the following command:

   $ sudo bash -c 'for i in {0..N}; do cpufreq-set -c $i -d <max_frequency> -u <max_frequency> -g <performance_mode_as_listed_by_cpufreq-info> done'

Note: Where N is the of CPUs to set, and max_frequency is the desired maximum CPU clock rate to set.

It may also be necessary to tune the network interface card (NIC) to eliminate drops (Ds) and reduce overflows (Os). This is done by increasing the number of RX descriptors (see Linux specific notes).