Difference between revisions of "Using Dual 10 Gigabit Ethernet on the USRP X300/X310"

From Ettus Knowledge Base
Jump to: navigation, search
(Add load balancing info)
(tweak wording)
 
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
==Application Note Number==
 
==Application Note Number==
 +
 
'''AN-296'''
 
'''AN-296'''
 +
<!-- for formatting purposes, keep these comments without extra space between them -->
 
<!-- Internal use only: please do keep this updated!
 
<!-- Internal use only: please do keep this updated!
 
==Revision History==
 
==Revision History==
Line 15: Line 17:
 
|style="text-align:center;"| Michael Dickens
 
|style="text-align:center;"| Michael Dickens
 
|style="text-align:center;"| Make this section a comment; Add info about load balancing args
 
|style="text-align:center;"| Make this section a comment; Add info about load balancing args
 +
|-
 +
|style="text-align:center;"| 2020-05-21
 +
|style="text-align:center;"| Michael Dickens
 +
|style="text-align:center;"| Whitespace & fix "benchmark_rate" to start with "./" and note why
 
|} -->
 
|} -->
  
 
==Abstract==
 
==Abstract==
 +
 
This short guide is meant to help in quickly setting up an X-series USRP for use over two 10 Gigabit Ethernet links simultaneously.
 
This short guide is meant to help in quickly setting up an X-series USRP for use over two 10 Gigabit Ethernet links simultaneously.
  
 
==Overview==
 
==Overview==
 +
 
''Disclaimer: In order to run at full receive rate (200 Msps) on two channels, it is recommended to have a multicore machine with clock frequencies above 3 GHz per core.''
 
''Disclaimer: In order to run at full receive rate (200 Msps) on two channels, it is recommended to have a multicore machine with clock frequencies above 3 GHz per core.''
  
Line 26: Line 34:
  
 
==Build Steps==
 
==Build Steps==
 +
 
Grab the latest UHD master by running the following commands:
 
Grab the latest UHD master by running the following commands:
  
Line 33: Line 42:
 
     $ make test
 
     $ make test
 
     $ make install
 
     $ make install
 
  
 
Next, build the latest XG image:
 
Next, build the latest XG image:
Line 42: Line 50:
  
 
OR
 
OR
 +
 
     $ make X310_XG
 
     $ make X310_XG
  
Line 47: Line 56:
  
 
==Device Setup==
 
==Device Setup==
 +
 
Ensure that the device is available for communication by pinging its initial address (usually <code>192.168.10.2</code> on the HG/S image).
 
Ensure that the device is available for communication by pinging its initial address (usually <code>192.168.10.2</code> on the HG/S image).
 
In this step, the XG image will be burned into memory for the X300 or X310 FPGA:
 
In this step, the XG image will be burned into memory for the X300 or X310 FPGA:
Line 70: Line 80:
 
     $ sudo ethtool -G <interface eth0, eth1, etc.> rx <max RX from previous command>
 
     $ sudo ethtool -G <interface eth0, eth1, etc.> rx <max RX from previous command>
  
To test the installation and USRP, try running benchmark_rate:
+
To test the installation and USRP, try running benchmark_rate, which can be done using the whole path to the command or from the directory in which it is installed; we use the latter here.
  
     $ benchmark_rate --args="type=x300,addr=<First IP, typically 192.168.30.2>,second_addr=<Second IP, typically 192.168.40.2>" --channels 0,1 --rx_rate 200e6 --duration 30
+
     $ ./benchmark_rate --args="type=x300,addr=<First IP, typically 192.168.30.2>,second_addr=<Second IP, typically 192.168.40.2>" --channels 0,1 --rx_rate 200e6 --duration 30
  
''Note: the order of the provided IPs should not matter for testing. The front panel LEDs can be used for verifying IP/channel mapping.''
+
''Note:'' the order of the provided IPs should not matter for testing. The front panel LEDs can be used for verifying IP/channel mapping.
  
''Note: When using UHD 3.15 or prior, only the <First IP> link will be used by default. To enable load balancing using both links, you need to augment the "--args" section to include "enable_tx_dual_eth=1,skip_dram=1"; thus the commandline could be:
+
''Note:'' When using UHD 3.15 or prior, only the <First IP> link will be used by default for TX only; both links are used for RX by default. If the aggregate TX data rate is more than can be provided by one link, then there are special flags to add to the "--args" section to support dual-link data transport ('load balancing' of a sort): "enable_tx_dual_eth=1,skip_dram=1". If the aggregate TX data rate is less than that provided by one link, then the primary link will be used regardless of these special flags. A commandline using these special flags could be:
  
     $ benchmark_rate --args="type=x300,addr=<First IP, typically 192.168.30.2>,second_addr=<Second IP, typically 192.168.40.2>,enable_tx_dual_eth=1,skip_dram=1" --channels 0,1 --rx_rate 200e6 --duration 30
+
     $ ./benchmark_rate --args="type=x300,addr=<First IP, typically 192.168.30.2>,second_addr=<Second IP, typically 192.168.40.2>,enable_tx_dual_eth=1,skip_dram=1" --channels 0,1 --rx_rate 200e6 --duration 30
  
 
==Host Tuning==
 
==Host Tuning==
 +
 
If overflows are encountered (or drops if any buffers are overwhelmed), the CPU might be throttling. This can be checked with <code>cpufreq</code> or <code>cpupower</code>.
 
If overflows are encountered (or drops if any buffers are overwhelmed), the CPU might be throttling. This can be checked with <code>cpufreq</code> or <code>cpupower</code>.
  
Line 95: Line 106:
 
'''Note:''' Where <code>N</code> is the of CPUs to set, and <code>max_frequency</code> is the desired maximum CPU clock rate to set.
 
'''Note:''' Where <code>N</code> is the of CPUs to set, and <code>max_frequency</code> is the desired maximum CPU clock rate to set.
  
 
+
It may also be necessary to tune the network interface card (NIC) to eliminate drops (Ds) and reduce overflows (Os). This is done by increasing the number of RX descriptors ([http://files.ettus.com/manual/page_transport.html#transport_udp_linux see Linux specific notes]).
 
+
 
+
  
 
[[Category:Application Notes]]
 
[[Category:Application Notes]]

Latest revision as of 09:49, 21 May 2020

Application Note Number

AN-296

Abstract

This short guide is meant to help in quickly setting up an X-series USRP for use over two 10 Gigabit Ethernet links simultaneously.

Overview

Disclaimer: In order to run at full receive rate (200 Msps) on two channels, it is recommended to have a multicore machine with clock frequencies above 3 GHz per core.

Ettus recommends using the Intel Ethernet Converged Network Adapter X520-DA2 for X-series USRPs over ethernet. This guide also assumes the user has the required software (Xilinx tools such as Vivado) to build the X-series FPGA image.

Build Steps

Grab the latest UHD master by running the following commands:

   $ git clone https://github.com/EttusResearch/uhd.git
   $ cd uhd/host && mkdir build
   $ cmake ../
   $ make test
   $ make install

Next, build the latest XG image:

   $ git clone https://github.com/EttusResearch/fpga.git
   $ cd fpga/usrp3/top/x300
   $ make X300_XG 

OR

   $ make X310_XG

This puts an image in build-X3XX_XG.

Device Setup

Ensure that the device is available for communication by pinging its initial address (usually 192.168.10.2 on the HG/S image). In this step, the XG image will be burned into memory for the X300 or X310 FPGA:

   $ uhd_image_loader --args="type=x300,addr=<IP ADDR>" --fpga-path="<path to the image we just built with .bit extension>"
  • Note: make sure your image matches your X-series USRP, as the sizes are different between the X300 and X310.*

We should always have the MTU size above 8000 (make it 9000) for both 10GigE interfaces. This can usually be done with ifconfig:

   $ sudo ifconfig <interface eth0, eth1, etc.> mtu 9000

Make sure you can find both USRP links with dual 10GE and ping the device on both IPs (try 192.168.30.2 and 192.168.40.2 by default).

Issues can arise with the NIC buffering, so it is a good idea to set the maximum number of descriptors with ethtool.

In order to get the maximum number of descriptors, run the following:

   $ sudo ethtool -g <interface>

To set the number of descriptors:

   $ sudo ethtool -G <interface eth0, eth1, etc.> rx <max RX from previous command>

To test the installation and USRP, try running benchmark_rate, which can be done using the whole path to the command or from the directory in which it is installed; we use the latter here.

   $ ./benchmark_rate --args="type=x300,addr=<First IP, typically 192.168.30.2>,second_addr=<Second IP, typically 192.168.40.2>" --channels 0,1 --rx_rate 200e6 --duration 30

Note: the order of the provided IPs should not matter for testing. The front panel LEDs can be used for verifying IP/channel mapping.

Note: When using UHD 3.15 or prior, only the <First IP> link will be used by default for TX only; both links are used for RX by default. If the aggregate TX data rate is more than can be provided by one link, then there are special flags to add to the "--args" section to support dual-link data transport ('load balancing' of a sort): "enable_tx_dual_eth=1,skip_dram=1". If the aggregate TX data rate is less than that provided by one link, then the primary link will be used regardless of these special flags. A commandline using these special flags could be:

   $ ./benchmark_rate --args="type=x300,addr=<First IP, typically 192.168.30.2>,second_addr=<Second IP, typically 192.168.40.2>,enable_tx_dual_eth=1,skip_dram=1" --channels 0,1 --rx_rate 200e6 --duration 30

Host Tuning

If overflows are encountered (or drops if any buffers are overwhelmed), the CPU might be throttling. This can be checked with cpufreq or cpupower.

As an example, consider running cpufreq:

   $ cpufreq-info

This command tells us the current state and min/max frequency of the CPU cores.

To set the maximum, run the following command:

   $ sudo bash -c 'for i in {0..N}; do cpufreq-set -c $i -d <max_frequency> -u <max_frequency> -g <performance_mode_as_listed_by_cpufreq-info> done'

Note: Where N is the of CPUs to set, and max_frequency is the desired maximum CPU clock rate to set.

It may also be necessary to tune the network interface card (NIC) to eliminate drops (Ds) and reduce overflows (Os). This is done by increasing the number of RX descriptors (see Linux specific notes).