Difference between revisions of "USRP Host Performance Tuning Tips and Tricks"

From Ettus Knowledge Base
Jump to: navigation, search
(Disable Hyper-threading)
(Disable Hyper-threading: update with modern language)
 
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
==Application Note Number==
 
==Application Note Number==
 
'''AN-088'''
 
'''AN-088'''
 
==Revision History==
 
{| class="wikitable"
 
!Date
 
!Author
 
!Details
 
|-
 
|style="text-align:center;"| 2019-06-19
 
|style="text-align:center;"| Nate Temple
 
|style="text-align:center;"| Initial creation
 
|}
 
  
 
==Overview==
 
==Overview==
Line 29: Line 18:
 
To set the CPU governor to <code>performance</code> for all cores:
 
To set the CPU governor to <code>performance</code> for all cores:
  
     for ((i=0;i<$(nproc);i++)); do sudo cpufreq-set -c $i -r -g performance; done
+
     for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done
  
 
You can then verify that the CPU governor has been set by running the command:
 
You can then verify that the CPU governor has been set by running the command:
Line 63: Line 52:
  
 
==Adjust Network Buffers==
 
==Adjust Network Buffers==
This applies to Ethernet connected USRPs (N2xx, N3xx, X3xx, E320).
+
This applies to USRP devices connected via Ethernet, such as the N200, N210, N300, N310, N320, N321, X300, X310, E320.
 
+
===N2xx===
+
    sudo sysctl -w net.core.rmem_max=625000000
+
    sudo sysctl -w net.core.wmem_max=625000000
+
  
===X3xx/N3xx/E320===
+
Note that these settings will not persist across a reboot.
  
     sudo sysctl -w net.core.wmem_max=576000
+
     sudo sysctl -w net.core.wmem_max=33554432
 
     sudo sysctl -w net.core.rmem_max=33554432
 
     sudo sysctl -w net.core.rmem_max=33554432
 +
    sudo sysctl -w net.core.wmem_default=33554432
 +
    sudo sysctl -w net.core.rmem_default=33554432
  
 
==Adjust Ethernet MTU==
 
==Adjust Ethernet MTU==
Line 79: Line 66:
 
For 1 Gigabit connections, the MTU should be set to <code>1500</code>.
 
For 1 Gigabit connections, the MTU should be set to <code>1500</code>.
  
For 10 Gigabit connections, the MTU should be set to <code>8000</code>.
+
For 10 Gigabit connections, the MTU should be set to <code>9000</code>.
  
It is important to set the value and not leave it is <code>automatic</code>
+
It is important to set the value and '''not''' leave it is <code>automatic</code>
  
 
==Increasing Ring Buffers==
 
==Increasing Ring Buffers==
Line 93: Line 80:
 
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html  
 
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html  
  
==Disable Hyper-threading==
+
== Disable Hyper-threading ==
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.
+
 
 +
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.
  
== Disable KPTI Protections for Spectra/Meltdown==
+
== Disable KPTI Protections for Spectre/Meltdown==
 
In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.
 
In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.
  
Line 102: Line 90:
 
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)
 
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)
  
Disabling KPTI protections can be done by adding the lines below to your `/etc/default/grub` file at `GRUB_CMDLINE_LINUX_DEFAULT=""`
+
Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code>
  
 
     pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier
 
     pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier
  
After modifying the `grub` file, run the following command to update your configuration and reboot:
+
After modifying the <code>grub</code> file, run the following command to update your configuration and reboot:
  
 
     sudo update-grub
 
     sudo update-grub

Latest revision as of 14:20, 21 November 2023

Application Note Number

AN-088

Overview

This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.

CPU Governor

Ensure your CPU governor is set to performance. This can be done with the Linux utility cpufrequtils.

Install cpufrequtils with the command below:

   sudo apt install cpufrequtils

You can then set the CPU governor to performance per core by issuing the command:

   sudo cpufreq-set -c $core_number -g performance

To set the CPU governor to performance for all cores:

   for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done

You can then verify that the CPU governor has been set by running the command:

   cpufreq-info

Thread Priority Scheduling

When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.

UHD Warning:
    Unable to set the thread priority. Performance may be negatively affected.
    Please see the general application notes in the manual for instructions.
    EnvironmentError: OSError: error in pthread_setschedparam

To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group usrp, adding your user to it, and then appending the line @usrp - rtprio 99 to the file /etc/security/limits.conf.

   sudo groupadd usrp
   sudo usermod -aG usrp $USER

Then add the line below to end of the file /etc/security/limits.conf:

   @usrp - rtprio  99

You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the /etc/group file.

There is further documentation about this in the User Manual at the link below.


Adjust Network Buffers

This applies to USRP devices connected via Ethernet, such as the N200, N210, N300, N310, N320, N321, X300, X310, E320.

Note that these settings will not persist across a reboot.

   sudo sysctl -w net.core.wmem_max=33554432
   sudo sysctl -w net.core.rmem_max=33554432
   sudo sysctl -w net.core.wmem_default=33554432
   sudo sysctl -w net.core.rmem_default=33554432

Adjust Ethernet MTU

This applies to Ethernet connected USRPs (N2xx, N3xx, X3xx, E320).

For 1 Gigabit connections, the MTU should be set to 1500.

For 10 Gigabit connections, the MTU should be set to 9000.

It is important to set the value and not leave it is automatic

Increasing Ring Buffers

This applies to Ethernet connected USRPs using a 10 Gb interface (X3xx, N3xx, E320).

Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates.

   sudo ethtool -G <interface> tx 4096 rx 4096

DPDK

DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html

Disable Hyper-threading

In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.

Disable KPTI Protections for Spectre/Meltdown

In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.

Disabling KPTI protections can be done by adding the lines below to your /etc/default/grub file at GRUB_CMDLINE_LINUX_DEFAULT=""

   pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier

After modifying the grub file, run the following command to update your configuration and reboot:

   sudo update-grub