Ettus Knowledge Base - User contributions [en]

Getting Started with DPDK and UHD

2026-09-08T12:18:54Z

MichaelDickens: /* UHD DPDK Version Dependencies */ add 4.11

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required. In other words, UHD + DPDK is generally for high-aggregate throughput use-cases, for example the X440 USRP running 8xTX or 8xRX at 500 MSps per channel or 2xTX or 2xRX at 2 GSps per channel. That said, "high aggregate" depends on the CPU and NIC; hence older CPUs may (and often does) require DPDK for the equivalent performance provided by modern CPUs without DPDK.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 and 4.11 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system with Intel CPU, this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

or

ifconfig

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed ''should'' be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Data Transport Issues on multi-CPU Systems using NUMA nodes ===

On multi-CPU systems, each CPU is a NUMA node. Even on a single CPU system one can intentionally allocate multiple NUMA nodes. A NUMA node contains specific CPU cores along with specific components such NICs and GPUs. For best performance with UHD, use the [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores <code>taskset</code>] to keep the UHD-based application within a given NUMA node's set of cores and make sure the NUMA node includes all NICs in use by UHD. Adding core isolation is ''not'' required when keeping execution within a specific NUMA node; it may be required for performance reasons.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:47:23Z

MichaelDickens: /* Abstract */ a little more depth

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required. In other words, UHD + DPDK is generally for high-aggregate throughput use-cases, for example the X440 USRP running 8xTX or 8xRX at 500 MSps per channel or 2xTX or 2xRX at 2 GSps per channel. That said, "high aggregate" depends on the CPU and NIC; hence older CPUs may (and often does) require DPDK for the equivalent performance provided by modern CPUs without DPDK.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system with Intel CPU, this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

or

ifconfig

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed ''should'' be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Data Transport Issues on multi-CPU Systems using NUMA nodes ===

On multi-CPU systems, each CPU is a NUMA node. Even on a single CPU system one can intentionally allocate multiple NUMA nodes. A NUMA node contains specific CPU cores along with specific components such NICs and GPUs. For best performance with UHD, use the [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores <code>taskset</code>] to keep the UHD-based application within a given NUMA node's set of cores and make sure the NUMA node includes all NICs in use by UHD. Adding core isolation is ''not'' required when keeping execution within a specific NUMA node; it may be required for performance reasons.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:41:46Z

MichaelDickens: /* Regular Data Transport Issues on multi-CPU Systems using NUMA nodes */ link taskset to that section

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system with Intel CPU, this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

or

ifconfig

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed ''should'' be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Data Transport Issues on multi-CPU Systems using NUMA nodes ===

On multi-CPU systems, each CPU is a NUMA node. Even on a single CPU system one can intentionally allocate multiple NUMA nodes. A NUMA node contains specific CPU cores along with specific components such NICs and GPUs. For best performance with UHD, use the [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores <code>taskset</code>] to keep the UHD-based application within a given NUMA node's set of cores and make sure the NUMA node includes all NICs in use by UHD. Adding core isolation is ''not'' required when keeping execution within a specific NUMA node; it may be required for performance reasons.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:40:53Z

MichaelDickens: /* Regular Data Transport Issue on multi-CPU Systems */ clarifying

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system with Intel CPU, this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

or

ifconfig

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed ''should'' be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Data Transport Issues on multi-CPU Systems using NUMA nodes ===

On multi-CPU systems, each CPU is a NUMA node. Even on a single CPU system one can intentionally allocate multiple NUMA nodes. A NUMA node contains specific CPU cores along with specific components such NICs and GPUs. For best performance with UHD, use the <code>taskset</code> to keep the UHD-based application within a given NUMA node's set of cores and make sure the NUMA node includes all NICs in use by UHD. Adding core isolation is ''not'' required when keeping execution within a specific NUMA node; it may be required for performance reasons.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:37:59Z

MichaelDickens: /* Regular Overflows on multi-CPU Systems */ clarifying

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system with Intel CPU, this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

or

ifconfig

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed ''should'' be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Data Transport Issue on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU, optimally along with PCIe devices such NICs. For best performance with UHD and NUMA nodes, use the <code>taskset</code> to keep the UHD-based application within a given NUMA node set of cores and make sure the NUMA node includes all NICs in use by UHD. Adding core isolation is not required when keeping execution within a specific NUMA node; it may be required for performance reasons.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:24:22Z

MichaelDickens: /* UHD 4.x */ remove redundant note

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system with Intel CPU, this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

or

ifconfig

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed ''should'' be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:22:49Z

MichaelDickens: /* Preparing your UHD Configuration File */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system with Intel CPU, this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

or

ifconfig

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:22:11Z

MichaelDickens: /* Enable hugepages */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system with Intel CPU, this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:21:48Z

MichaelDickens: /* Enable hugepages */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. The following is an easy way to enable this feature and allocate enough <code>hugepages</code> For UHD's DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

hugepages=2048

On a vanilla Ubuntu system this entry may read like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:18:36Z

MichaelDickens: /* USRPs */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links, whether physical or virtual:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:17:43Z

MichaelDickens: /* Abstract */ rewording

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. DPDK is intended for use-cases where standard networking is not performant enough; the dedicated resources are otherwise wasted if not required.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T20:15:53Z

MichaelDickens: /* Abstract */ general tweaks

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

== Abstract ==

UHD's original support for networked devices used the Linux kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace the UHD thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple cores, the timing-critical threads might sporadically be hit with sleeping time and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes caused by these make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user-space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the Linux thread scheduler. In addition, DPDK lowers the overall overhead for packet processing.

DPDK utilizes dedicated resources -- CPU cores and memory -- to increase data throughput. This tradeoff may be acceptable for some hardware and some use-cases. Using DPDK should generally be limited to cases where standard networking is not performant enough.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T19:54:17Z

MichaelDickens: update history

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T19:31:23Z

MichaelDickens: whitespace

== Application Note Number ==
'''AN-088'''


== Overview ==

This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 GbE links, the MTU should be set to <code>1500</code>.

For 10 GbE and 100 GbE links, the MTU should be set to <code>9000</code>.

It is important to ''set'' the value and ''not'' leave it in the default: <code>automatic</code>

== Increasing Ring Buffers ==

This applies to Ethernet connected USRPs using a 10 GbE or 100 GbE interface: N2xx, N3xx, X3xx, E320, X4xx

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates. For example:

sudo ethtool -G <interface> tx 4096 rx 4096

To see the NIC limits, use

sudo ethtool -g <interface>

== DPDK ==

DPDK is supported on USRPs with 10 Gbe (SFP+) and 100 GbE (QSFP28) links. DPDK replaces the traditional Linux networking stack and NIC driver with a low overhead user-land based driver. See the [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD DPDK AppNote] for specifics.

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown ==

In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>.

For Linux kernel 5:

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

For Linux kernel 6 or higher:

mitigations=off

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T19:30:57Z

MichaelDickens: /* Application Note Number */ add history comment

==Application Note Number==
'''AN-088'''



==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 GbE links, the MTU should be set to <code>1500</code>.

For 10 GbE and 100 GbE links, the MTU should be set to <code>9000</code>.

It is important to ''set'' the value and ''not'' leave it in the default: <code>automatic</code>

== Increasing Ring Buffers ==

This applies to Ethernet connected USRPs using a 10 GbE or 100 GbE interface: N2xx, N3xx, X3xx, E320, X4xx

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates. For example:

sudo ethtool -G <interface> tx 4096 rx 4096

To see the NIC limits, use

sudo ethtool -g <interface>

== DPDK ==

DPDK is supported on USRPs with 10 Gbe (SFP+) and 100 GbE (QSFP28) links. DPDK replaces the traditional Linux networking stack and NIC driver with a low overhead user-land based driver. See the [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD DPDK AppNote] for specifics.

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown ==

In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>.

For Linux kernel 5:

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

For Linux kernel 6 or higher:

mitigations=off

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T19:05:34Z

MichaelDickens: /* DPDK */ change link to KB AN

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 GbE links, the MTU should be set to <code>1500</code>.

For 10 GbE and 100 GbE links, the MTU should be set to <code>9000</code>.

It is important to ''set'' the value and ''not'' leave it in the default: <code>automatic</code>

== Increasing Ring Buffers ==

This applies to Ethernet connected USRPs using a 10 GbE or 100 GbE interface: N2xx, N3xx, X3xx, E320, X4xx

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates. For example:

sudo ethtool -G <interface> tx 4096 rx 4096

To see the NIC limits, use

sudo ethtool -g <interface>

== DPDK ==

DPDK is supported on USRPs with 10 Gbe (SFP+) and 100 GbE (QSFP28) links. DPDK replaces the traditional Linux networking stack and NIC driver with a low overhead user-land based driver. See the [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD DPDK AppNote] for specifics.

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown ==

In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>.

For Linux kernel 5:

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

For Linux kernel 6 or higher:

mitigations=off

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T19:04:17Z

MichaelDickens: /* DPDK */

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 GbE links, the MTU should be set to <code>1500</code>.

For 10 GbE and 100 GbE links, the MTU should be set to <code>9000</code>.

It is important to ''set'' the value and ''not'' leave it in the default: <code>automatic</code>

== Increasing Ring Buffers ==

This applies to Ethernet connected USRPs using a 10 GbE or 100 GbE interface: N2xx, N3xx, X3xx, E320, X4xx

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates. For example:

sudo ethtool -G <interface> tx 4096 rx 4096

To see the NIC limits, use

sudo ethtool -g <interface>

== DPDK ==

DPDK is supported on USRPs with 10 Gbe (SFP+) and 100 GbE (QSFP28) links. DPDK replaces the traditional Linux networking stack and NIC driver with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown ==

In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>.

For Linux kernel 5:

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

For Linux kernel 6 or higher:

mitigations=off

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T19:03:19Z

MichaelDickens: /* Adjust Ethernet MTU */

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 GbE links, the MTU should be set to <code>1500</code>.

For 10 GbE and 100 GbE links, the MTU should be set to <code>9000</code>.

It is important to ''set'' the value and ''not'' leave it in the default: <code>automatic</code>

== Increasing Ring Buffers ==

This applies to Ethernet connected USRPs using a 10 GbE or 100 GbE interface: N2xx, N3xx, X3xx, E320, X4xx

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates. For example:

sudo ethtool -G <interface> tx 4096 rx 4096

To see the NIC limits, use

sudo ethtool -g <interface>

== DPDK ==
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown ==

In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>.

For Linux kernel 5:

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

For Linux kernel 6 or higher:

mitigations=off

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T19:01:01Z

MichaelDickens: /* USRPs */ more concise pagination

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices, which are those providing 10 GbE (SFP+) and 100 GbE (QSFP28) links:

E320, N300 / N310, N320 / N321, X300 / X310, X410, X420, X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:59:14Z

MichaelDickens: /* UHD DPDK Version Dependencies */ remove NOTE because it's the next section anyway

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:58:51Z

MichaelDickens: /* UHD DPDK Version Dependencies */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 and 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

'''NOTE:''' DPDK 19 and newer is easy to download and install from source. If doing so, UHD ''must'' also be built from source. In this case, we ''strongly'' recommend using the latest DPDK version supported by the UHD version being built from source.

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:58:20Z

MichaelDickens: /* References */ reduce and make generic ... fewer updates in the future!

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
* DPDK Docs by Version: https://core.dpdk.org/doc/archives/
* DPDK Quick Install Guide: https://core.dpdk.org/doc/quick-start/
* UHD Manual on DPDK: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

'''NOTE:''' DPDK 19 and newer is easy to download and install from source. If doing so, UHD ''must'' also be built from source. In this case, we ''strongly'' recommend using the latest DPDK version supported by the UHD version being built from source.

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:55:18Z

MichaelDickens: /* Dependencies */ tweak title

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== UHD DPDK Version Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

'''NOTE:''' DPDK 19 and newer is easy to download and install from source. If doing so, UHD ''must'' also be built from source. In this case, we ''strongly'' recommend using the latest DPDK version supported by the UHD version being built from source.

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:54:32Z

MichaelDickens: /* Installing DPDK */ elaborate on UHD + DPDK for updated NIC drivers

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

'''NOTE:''' DPDK 19 and newer is easy to download and install from source. If doing so, UHD ''must'' also be built from source. In this case, we ''strongly'' recommend using the latest DPDK version supported by the UHD version being built from source.

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

'''NOTE:''' It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work (e.g., the system provided NIC drivers are too old). In this case, to use DPDK one ''must'' build DPDK and UHD from source. Follow these steps in the order noted to guarantee that DPDK is built with the updated NIC drivers and that UHD is built with the noted DPDK:

# Remove the system provided DPDK and DPDK-DEV and all UHD install(s)
# Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed
# Download and install DPDK from source
# Download and install UHD from source

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:51:02Z

MichaelDickens: /* Installing DPDK */ tweaks

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

'''NOTE:''' DPDK 19 and newer is easy to download and install from source. If doing so, UHD ''must'' also be built from source. In this case, we ''strongly'' recommend using the latest DPDK version supported by the UHD version being built from source.

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK version 18 and older can be challenging to ''correctly'' build from source. DPDK version 19 and newer use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release ("long term service" or LTS). While any ''can'' be used with UHD, we ''strongly'' recommend using just the formal releases.

NOTE: It is sometimes necessary to manually install NIC device drivers because of improved feature sets or just getting a NIC to work, or sometimes the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:48:00Z

MichaelDickens: /* Dependencies */ add NOTE

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

'''NOTE:''' DPDK 19 and newer is easy to download and install from source. If doing so, UHD ''must'' also be built from source. In this case, we ''strongly'' recommend using the latest DPDK version supported by the UHD version being built from source.

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T18:45:40Z

MichaelDickens: /* Disable KPTI Protections for Spectre/Meltdown */ tweak note to be consistent with other same notes

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 Gigabit connections, the MTU should be set to <code>1500</code>.

For 10 Gigabit connections, the MTU should be set to <code>9000</code>.

It is important to set the value and '''not''' leave it is <code>automatic</code>.

== Increasing Ring Buffers ==

This applies to Ethernet connected USRPs using a 10 GbE or 100 GbE interface: N2xx, N3xx, X3xx, E320, X4xx

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates. For example:

sudo ethtool -G <interface> tx 4096 rx 4096

To see the NIC limits, use

sudo ethtool -g <interface>

== DPDK ==
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown ==

In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>.

For Linux kernel 5:

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

For Linux kernel 6 or higher:

mitigations=off

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T18:44:49Z

MichaelDickens: /* Disable KPTI Protections for Spectre/Meltdown */ kernel 5 versus 6

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 Gigabit connections, the MTU should be set to <code>1500</code>.

For 10 Gigabit connections, the MTU should be set to <code>9000</code>.

It is important to set the value and '''not''' leave it is <code>automatic</code>.

== Increasing Ring Buffers ==

This applies to Ethernet connected USRPs using a 10 GbE or 100 GbE interface: N2xx, N3xx, X3xx, E320, X4xx

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates. For example:

sudo ethtool -G <interface> tx 4096 rx 4096

To see the NIC limits, use

sudo ethtool -g <interface>

== DPDK ==
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown ==

In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>.

For Linux kernel 5:

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

For Linux kernel 6 or higher:

mitigations=off

'''NOTE:''' After modifying the <code>grub</code> file, run the following command to update your configuration and reboot:

sudo update-grub

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T18:43:30Z

MichaelDickens: /* Increasing Ring Buffers */ NIC limits & exhaustive compressed USRP list

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 Gigabit connections, the MTU should be set to <code>1500</code>.

For 10 Gigabit connections, the MTU should be set to <code>9000</code>.

It is important to set the value and '''not''' leave it is <code>automatic</code>.

== Increasing Ring Buffers ==

This applies to Ethernet connected USRPs using a 10 GbE or 100 GbE interface: N2xx, N3xx, X3xx, E320, X4xx

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates. For example:

sudo ethtool -G <interface> tx 4096 rx 4096

To see the NIC limits, use

sudo ethtool -g <interface>

== DPDK ==
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown==
In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

After modifying the <code>grub</code> file, run the following command to update your configuration and reboot:

sudo update-grub

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T18:41:36Z

MichaelDickens: /* Adjust Network Buffers */ compressed exhaustive list

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N2xx, N3xx, X3xx, E320, X4xx

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 Gigabit connections, the MTU should be set to <code>1500</code>.

For 10 Gigabit connections, the MTU should be set to <code>9000</code>.

It is important to set the value and '''not''' leave it is <code>automatic</code>.

==Increasing Ring Buffers==
This applies to Ethernet connected USRPs using a 10 Gb interface (X3xx, N3xx, E320).

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates.

sudo ethtool -G <interface> tx 4096 rx 4096

== DPDK ==
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown==
In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

After modifying the <code>grub</code> file, run the following command to update your configuration and reboot:

sudo update-grub

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T18:41:10Z

MichaelDickens: /* Adjust Ethernet MTU */ exhaustive USRP list

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N200, N210, N300, N310, N320, N321, X300, X310, E320, X410, X420, and X440

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

== Adjust Ethernet MTU ==

This applies to Ethernet connected USRPs: N2xx, N3xx, X3xx, E31x, E320, X4xx

For 1 Gigabit connections, the MTU should be set to <code>1500</code>.

For 10 Gigabit connections, the MTU should be set to <code>9000</code>.

It is important to set the value and '''not''' leave it is <code>automatic</code>.

==Increasing Ring Buffers==
This applies to Ethernet connected USRPs using a 10 Gb interface (X3xx, N3xx, E320).

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates.

sudo ethtool -G <interface> tx 4096 rx 4096

== DPDK ==
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown==
In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

After modifying the <code>grub</code> file, run the following command to update your configuration and reboot:

sudo update-grub

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T18:40:08Z

MichaelDickens: /* Adjust Network Buffers */ exhaustive list of ENET based USRPs. Add larger setting

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==

This applies to USRP devices connected via 10 GbE or 100 GbE Ethernet: N200, N210, N300, N310, N320, N321, X300, X310, E320, X410, X420, and X440

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

'''NOTE:''' You may choose to use a larger setting such as 250000000 to guarantee these settings work for ''any'' USRP. Larger numbers require memory allocation but with modern hardware this memory is generally available with plenty of spare.

==Adjust Ethernet MTU==
This applies to Ethernet connected USRPs (N2xx, N3xx, X3xx, E320).

For 1 Gigabit connections, the MTU should be set to <code>1500</code>.

For 10 Gigabit connections, the MTU should be set to <code>9000</code>.

It is important to set the value and '''not''' leave it is <code>automatic</code>

==Increasing Ring Buffers==
This applies to Ethernet connected USRPs using a 10 Gb interface (X3xx, N3xx, E320).

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates.

sudo ethtool -G <interface> tx 4096 rx 4096

== DPDK ==
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown==
In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

After modifying the <code>grub</code> file, run the following command to update your configuration and reboot:

sudo update-grub

[[Category:Application Notes]]

USRP Host Performance Tuning Tips and Tricks

2026-08-30T18:36:10Z

MichaelDickens: /* CPU Governor */ add note about cpupower

==Application Note Number==
'''AN-088'''

==Overview==
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

== CPU Governor ==

'''NOTE:''' use of <code>cpufrequtils</code> noted below ''should'' be limited to Linux kernel 5. The utility <code>cpupower</code> should be used in Linux kernel 6 and newer.

Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.

Install <code>cpufrequtils</code> with the command below:

sudo apt install cpufrequtils

You can then set the CPU governor to <code>performance</code> per core by issuing the command:

sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to <code>performance</code> for all cores:

for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

cpufreq-info

==Thread Priority Scheduling==

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

<pre>
UHD Warning:
Unable to set the thread priority. Performance may be negatively affected.
Please see the general application notes in the manual for instructions.
EnvironmentError: OSError: error in pthread_setschedparam
</pre>

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.

sudo groupadd usrp
sudo usermod -aG usrp $USER

Then add the line below to end of the file <code>/etc/security/limits.conf</code>:

@usrp - rtprio 99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.

There is further documentation about this in the User Manual at the link below.

* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]

==Adjust Network Buffers==
This applies to USRP devices connected via Ethernet, such as the N200, N210, N300, N310, N320, N321, X300, X310, E320, X410, X440.

Note that these settings will not persist across a reboot.

sudo sysctl -w net.core.wmem_max=33554432
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_default=33554432

To make these settings persistent, set or add the following in <code>/etc/sysctl.conf</code>:

net.core.wmem_max=33554432
net.core.rmem_max=33554432
net.core.wmem_default=33554432
net.core.rmem_default=33554432

==Adjust Ethernet MTU==
This applies to Ethernet connected USRPs (N2xx, N3xx, X3xx, E320).

For 1 Gigabit connections, the MTU should be set to <code>1500</code>.

For 10 Gigabit connections, the MTU should be set to <code>9000</code>.

It is important to set the value and '''not''' leave it is <code>automatic</code>

==Increasing Ring Buffers==
This applies to Ethernet connected USRPs using a 10 Gb interface (X3xx, N3xx, E320).

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates.

sudo ethtool -G <interface> tx 4096 rx 4096

== DPDK ==
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

== Disable Hyper-threading ==

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

== Disable KPTI Protections for Spectre/Meltdown==
In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>

pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

After modifying the <code>grub</code> file, run the following command to update your configuration and reboot:

sudo update-grub

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:33:47Z

MichaelDickens: /* Increasing num_recv_frames */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for expected link data throughput of 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:33:11Z

MichaelDickens: /* Increasing num_recv_frames */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:32:33Z

MichaelDickens: /* Stopping Extraneous Processes */ add NOTE

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

'''NOTE:''' Stopping extraneous process, like disabling hyperthreading, should be a last resort. Using thoughtful core isolation and core process assignments (e.g., "taskset") should work the vast majority of the time.

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:29:41Z

MichaelDickens: /* Limit Execution CPUs/Cores */ add DPDK control core

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

'''NOTE:''' For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code> nor the hyperthreaded cores. The "DPDK control" core ''may'' be included in the taskset because it is not dedicated by DPDK for its sole use.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:28:34Z

MichaelDickens: /* Streaming on Multiple Channels using 1 Thread Per Stream */ add throughput note

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best ratio of throughput to number of threads or cores in use.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:27:32Z

MichaelDickens: /* Disable System Interrupts on Cores/CPUs */ add not about context switching

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable the vast majority of system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

'''NOTE:''' Adding these entries hugely reduces the interrupts being handled by the noted cores but do not prevent forced context switching to handle multiple threads per core.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best throughput.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:25:28Z

MichaelDickens: /* Disable System Interrupts on Cores/CPUs */ add note about core isolation and taskset

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

'''NOTE:''' Disabling system interrupts ''should'' be used in conjunction with core isolation, which ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Disabling system interrupts can improve the jitter and performance generally. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best throughput.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:24:08Z

MichaelDickens: /* Isolate Cores/CPUs */ add NOTE about using isolated cores

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

'''NOTE:''' Isolation cores ''must'' be used along with a [https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Limit_Execution_CPUs.2FCores taskset] or other method such that the UHD-based application uses the isolated cores.

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best throughput.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:21:00Z

MichaelDickens: /* Streaming on Multiple Channels using 1 Thread Per Stream */ add info about UHD-based application

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. For example, this can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code> -- which assigns 1 channel per thread, whether TX or RX.

With your own UHD-based application, you will need to determine how many channels to assign to a given thread / stream. It is not necessarily true that 1 channel per thread will provide the best throughput.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:17:18Z

MichaelDickens: /* Disable System Interrupts on Cores/CPUs */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4 and their hyperthreaded relations, add this entry:

nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:16:39Z

MichaelDickens: /* Isolate Cores/CPUs */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:

nohz_full=2-4 rcu_nocbs=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T18:16:00Z

MichaelDickens: /* Isolate Cores/CPUs */ much deeper info

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>" file). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

'''NOTE:''' After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.

If hyperthreading is enabled for the cores being isolated, also isolate the hyperthreaded cores. Note that some CPUs -- such as the Intel i7 and i9 -- have both performance and efficiency cores, and generally only the performance cores can be hyperthreaded. Thus, for example for an 8 physical core CPU with all cores hyperthreaded -- cores 0-7 as the physical cores and 8-15 are the hyperthreaded virtual cores with core 0 paired with hyperthreaded core 8, 1 paired with 9, and so forth -- this entry would be:

isolcpus=2-4,10-12

DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method.

With whatever cores are listed as isolated in GRUB, the Linux thread scheduler will ''generally'' not schedule threads or interrupts to run on them. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent forced context switching on threads running on those cores; these are handled by other GRUB parameters and/or system settings. In Linux kernel 6, better -- but still not full -- isolation would include these entries (for the same cores noted above):

isolcpus=nohz,domain,managed_irq,2-4,10-12 nohz_full=2-4,10-12 rcu_nocbs=2-4,10-12 rcu_nocb_poll

When determining the number of cores to isolate for UHD, take into account the number dedicated for DPDK: 1 core per DPDK link as of UHD 4.11. Thus for the possible 8-core CPU, if using 2 DPDK links there are just 6 cores remaining and one (core 0) must be dedicated for the Linux kernel -- and it's wise to dedicate 2 cores for Linux (cores 0 and 1) so that the system does not lock. In this 8-core CPU scenario if using 2 DPDK links, only 4 cores would be available for UHD. This may be enough cores for your UHD-based application; only you know the use-case and can thus determine how many cores the CPU should have to support it.

A good rule of thumb for UHD-based applications is 1 core for the general application, 1 core per RX link, and 2 core per TX link. While not 1:1 thread:core, the actual aggregate thread CPU% used maps well.

If the CPU has enough cores to support 1 thread per core and your application allows for this level of core isolation, then both isolating and assigning these cores will truly utilize the CPU for maximum UHD benefit.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:

nohz_full=2-4 rcu_nocbs=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T17:50:49Z

MichaelDickens: /* Increasing num_recv_frames */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>dpdk_num_mbufs</code> and sometimes <code>dpdk_mbuf_cache_size</code> and <code>dpdk_num_desc</code>, in the UHD DPDK configuration file.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent context switching; these are handled by other GRUB parameters and/or system settings.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:

nohz_full=2-4 rcu_nocbs=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T17:49:11Z

MichaelDickens: /* Increasing num_recv_frames */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where <code>###</.code> might be <code>500</code> for 10-20 Gbps and <code>3000</code> for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory allocation, which for DPDK means higher <code>num_buf</code> and sometimes <code>asd</code>.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent context switching; these are handled by other GRUB parameters and/or system settings.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:

nohz_full=2-4 rcu_nocbs=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T17:45:05Z

MichaelDickens: /* Increasing num_recv_frames */ Correct value info

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=###</code> can help, where ### might be 500 for 10-20 Gbps and 3000 for 80 Gbps. There is no "correct" value, unlike, for example, the number samples-per-packet which is bound by the Ethernet packet size: 1500 Bytes for 1 GbE links and 9000 Bytes ("jumbo packet size") for 10 GbE and 100 GbE links. Higher values require more memory.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent context switching; these are handled by other GRUB parameters and/or system settings.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:

nohz_full=2-4 rcu_nocbs=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-30T17:38:55Z

MichaelDickens: /* Running UHD Applications with DPDK */

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference is in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent context switching; these are handled by other GRUB parameters and/or system settings.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:

nohz_full=2-4 rcu_nocbs=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]

Getting Started with DPDK and UHD

2026-08-29T20:16:06Z

MichaelDickens: /* Additional Host Configuration for NIC Vendors */ update VFIO-PCI info

== Application Note Number and Authors ==

'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens


==Overview==

This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD.

==Abstract==

Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.

DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers.

== Supported Devices ==

=== USRPs ===

DPDK is supported on the following USRP devices:

* E320
* N300 / N310
* N320 / N321
* X300 / X310
* X410
* X420
* X440

=== Host Network Cards ===

DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.

* Intel X520-DA1 (1x10Gb)
* Intel X520-DA2 (2x10Gb)
* Intel X710-DA2 (2x10Gb)
* Intel X710-DA4 (4x10Gb)
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)
* Intel E810-CQDA1 (1x100Gb and 1x4x10Gb)
* Intel E810-CQDA2 (1x100Gb and 2x4x10Gb; [https://edc.intel.com/content/www/us/en/design/products/ethernet/perf-tuning-guide-800-series-linux/%E2%80%8Bcheck-system-hardware-capabilities/ note that this NIC ''does not'' support 2x100Gb])
* Intel E810-2CQDA2 (1x100Gb and 2x4x10Gb; 2x100Gb with some work)
* Intel E830-CQDA2 (2x100Gb or 8x10Gb)
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb)
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]

== References ==

* DPDK: https://www.dpdk.org/
** https://doc.dpdk.org/guides-17.11
** https://doc.dpdk.org/guides-18.11
** https://doc.dpdk.org/guides-19.11
** https://doc.dpdk.org/guides-20.11
** https://doc.dpdk.org/guides-21.11
** https://doc.dpdk.org/guides-22.11
** https://doc.dpdk.org/guides-23.11
** https://doc.dpdk.org/guides-24.11
** https://doc.dpdk.org/guides-25.11
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html

== Dependencies ==

* UHD 3.x requires DPDK 17.11

* UHD 4.0 and 4.1 require DPDK 18.11

* UHD 4.2 to 4.7 can use any version of DPDK from 18.11 to 21.11

* UHD 4.8 to 4.9 can use any version of DPDK from 18.11 to 24.11

* UHD 4.10 can use any version of DPDK from 19.11 to 25.11

== Installing DPDK ==

We recommend installing DPDK via the system-provided installer; for example with Ubuntu:

sudo apt install dpdk dpdk-dev

While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source though more recent versions use meson and ninja along with default settings needed by UHD and hence are relatively simple to build and install. If you require installing DPDK from source, the install guide for various versions is noted below:

* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]
* [https://doc.dpdk.org/guides-24.11/linux_gsg/build_dpdk.html DPDK 24.11]
* [https://doc.dpdk.org/guides-25.11/linux_gsg/build_dpdk.html DPDK 25.11]

DPDK releases come three times per year in April (version X.04), July (version X.07), and November (version X.11). The X.11 version is more of a formal release. While any ''can'' be used with UHD, we recommend using just the formal releases.

NOTE: It is sometimes necessary to use NIC device drivers because of improved feature sets or just getting a NIC to work; the system provided NIC drivers are too old. In this case: (1) remove the system provided DPDK and DPDK-DEV and whatever UHD install is in use; you will need to build these from source. (2) Download and install the new NIC drivers, NVM, and anything else that the NIC needs to be updated to the version needed. (3) Download and install DPDK from source. (4) Download and install UHD from source.

== Installing UHD ==

Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.

NOTE that in general UHD installed from PPA or system packages ''does not'' include support for DPDK, and even if DPDK is installed alongside these UHD it ''will not be used''. In order to get UHD with DPDK support, [https://kb.ettus.com/Building_and_Installing_the_USRP_Open-Source_Toolchain_(UHD_and_GNU_Radio)_on_Linux UHD ''must'' be built from source].

== Enable hugepages ==

DPDK ''requires'' <code>hugepages</code> to be enabled and "large enough". There are multiple ways to enable <code>hugepages</code>. THe following is an easy way to enable this feature and enough allocated for DPDK needs.

Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:

iommu=pt intel_iommu=on hugepages=2048

On a vanilla Ubuntu system it should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"

Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:

sudo update-grub

For these settings to take effect, reboot your host machine.

== Preparing your UHD Configuration File ==

You must note the MAC addresses for your NICs before proceeding.

The MAC addresses for your NICs can be found by running the command:

ip a

You must then create a UHD configuration file.

For UHD 3 the location is <code>/root/.uhd/uhd.conf</code>.

sudo su
mkdir -p /root/.uhd
nano /root/.uhd/uhd.conf

For UHD 4 the location is <code>/root/.config/uhd.conf</code>.

sudo su
mkdir -p /root/.config
nano /root/.config/uhd.conf

=== UHD 3.x ===

An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.

You should update the following fields for your configuration from this example:

* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC

* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.

* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)

* Update the <code>dpdk-ipv4</code> fields to your desired IP range.
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system

[use_dpdk=1]
dpdk-mtu=9000
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/
dpdk-corelist=2,3,4
dpdk-num-mbufs=4095
dpdk-mbufs-cache-size=315

[dpdk-mac=aa:bb:cc:dd:ee:f1]
dpdk-io-cpu = 3
dpdk-ipv4 = 192.168.10.1/24

[dpdk-mac=aa:bb:cc:dd:ee:f2]
dpdk-io-cpu = 4
dpdk-ipv4 = 192.168.20.1/24

'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config

=== UHD 4.x ===

An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.

You must verify and/or update the following fields for your configuration from this example:

* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s); note that the MAC address info ''must be lowercase''

* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.

* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).

* The <code>dpdk_ipv4</code> fields to your desired IP range(s).
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system
** <code>192.168.10.2</code> on a default E320 system
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4096
dpdk_num_desc=4096
dpdk_mbuf_cache_size=315

[dpdk_mac=aa:bb:cc:dd:ee:f1]
dpdk_lcore=3
dpdk_ipv4=192.168.10.1/24

[dpdk_mac=aa:bb:cc:dd:ee:f2]
dpdk_lcore=4
dpdk_ipv4=192.168.20.1/24

'''Notes:'''
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].
* The number of cores listed must be used exactly within the file.
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code>
* Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config

== Additional Host Configuration for NIC Vendors ==

The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.

For these NICs DPDK ''requires'' the <code>vfio-pci</code> driver, and it must generally be loaded beforehand:

sudo modprobe vfio-pci

=== Intel NICs ===

First, take the link(s) down; for example for the link name <code>eth1np1</code>:

sudo ifconfig eth1np1 down

'''Note:''' You ''may'' need to turn off the NIC first before doing the rebind. In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the link(s).

'''Note:''' Make sure to pick the correct link name(s) for your specific system! If there are multiple Intel NIC links to be used by DPDK take them all down, one each in turn.

Next, you will need to ''manually'' rebind the NIC to the <code>vfio-pci</code> drivers.

Identify the PCI address your NIC link(s) is/are at:

dpdk-devbind.py -s

Make note of the PCI address that your NIC link(s) is/are connected to for the next step; for example for this output:

0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=ixgbe unused=vfio-pci

one would use <code>02:00.0</code> and <code>02:00.1</code>, as follows:

sudo dpdk-devbind.py --bind=vfio-pci 02:00.0
sudo dpdk-devbind.py --bind=vfio-pci 02:00.1

'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.

'''Note:''' In older DPDK versions the command does not contain the trailing <code>.py</code> .

You should now see the NIC listed under DPDK devices; for example:

# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe

'''Note:''' More info can be found on the rebinding process [https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules here: https://doc.dpdk.org/guides-25.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules]

Once the NIC link(s) is/are binded to the <code>vfio-pci</code> driver they are ready for use by UHD.

=== Mellanox NICs ===

The NVIDIA Mellanox ("Mellanox") NICs ''do not'' require manual rebinding. Mellanox provides system IC drivers and additional drivers for DPDK to handle the binding to and from the <code>vfio-pci</code> driver.

In general the Mellanox system-provided NIC and DPDK drivers should work and you should not need to install Mellanox drivers. Mellanox provides tools for updating drivers and NIC firmware, if desired.

If for some reason your host OS does not provide Mellanox DPDK drivers, then these can be installed manually. For example:

sudo apt install librte-pmd-mlx5
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib

If for some reason you are running into issues using a Mellanox NIC, then you can [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ download and install the latest Mellanox drivers from the Mellanox website] (https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/).

The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 through -7 NICs. For DPDK 20 and older only this driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK. We recommend DPDK 21 and newer when possible; these versions are easy to build from source using <code>meson</code> and <code>ninja</code>.

== Running UHD Applications with DPDK ==

UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.

'''Important Note:''' In order for UHD to use DPDK, the UHD application ''must'' be ran as the <code>root</code> user; this can be done either by prepending <code>sudo</code> to the command or becoming the roort user via <code>sudo su</code>.

For example, running the <code>benchmark_rate</code> utility after issuing <code>sudo su</code> (one can also run this command simply as <code>sudo /usr/local/lib/uhd/examples/benchmark_rate ....</code>.

NOTE that the text output from UHD 3 or 4 is about the same; the primary difference will in the initial <code>[INFO]</code> line.

<pre>
# cd /usr/local/lib/uhd/examples

# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e
EAL: Detected 8 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: Ignore mapping IO port bar(2)
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down
EAL: Port 0 MAC: aa bb cc dd ee f1
EAL: Port 0 UP: 1
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down
EAL: Port 1 MAC: aa bb cc dd ee f2
EAL: Port 1 UP: 1
EAL: Init DONE!
EAL: Starting I/O threads!
USER2: Thread 1 started
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: ni-n3xx-313ABDA
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 0
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 0
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.728707] Setting device timestamp to 0...
[INFO] [MULTI_USRP] 1) catch time transition at pps edge
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels
[00:00:15.878071] Benchmark complete.

Benchmark rate summary:
Num received samples: 2557247854
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2504266704
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!
</pre>

== Tuning Notes ==

=== General Host Performance Tuning App Note ===

Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].

=== Increasing <code>num_recv_frames</code> ===

If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.

=== Full Rate Streaming (UHD 3.x only) ===

If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:

skip_ddc=1
skip_duc=1

=== Full Rate on X3xx ===

If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:

enable_tx_dual_eth=1

=== Isolate Cores/CPUs ===

Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:

isolcpus=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. DPDK handles core isolation during runtime for cores assigned to it, but it is OK to isolate cores used by DPDK via this method. Using just <code>isolcpus</code> does not ''entirely'' isolate cores, nor does it prevent context switching; these are handled by other GRUB parameters and/or system settings.

=== Disable System Interrupts on Cores/CPUs ===

Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:

nohz_full=2-4 rcu_nocbs=2-4

NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.

=== Streaming on Multiple Channels using 1 Thread Per Stream ===

If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.

=== Elevated Streaming Thread Priority ===

In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.

=== Extra <code>nice</code> Priority ===

Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:

sudo nice -n -20

=== Limit Execution CPUs/Cores ===

With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):

taskset -c "2-4"

NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.

=== Stopping Extraneous Processes ===

The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:

systemctl stop containerd containerd.socket
systemctl stop docker docker.socket
systemctl stop dbus dbus.socket
systemctl stop snapd snapd.socket
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket

The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:

systemctl stop gdm

=== Disable Hyper-threading ===

In some applications which require the highest possible CPU performance per core, disabling hyper-threading ''can'' provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

=== Additional Tuning Notes from Intel ===

* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html

== Known Issues / Troubleshooting ==

=== Regular Overflows on multi-CPU Systems ===

On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per the section <code> Isolate Cores/CPUs</code> above) make sure that all of the cores are in the same NUMA node.

=== Underruns Every Second with DPDK + Ubuntu ===

With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.

<pre>
$ sudo -s
$ cd /sys/kernel/debug/sched
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
NO_RT_RUNTIME_SHARE
$ echo RT_RUNTIME_SHARE > features
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE
RT_RUNTIME_SHARE
</pre>

[[Category:Application Notes]]