https://kb.ettus.com/api.php?action=feedcontributions&user=MichaelDickens&feedformat=atomEttus Knowledge Base - User contributions [en]2024-03-28T22:07:56ZUser contributionsMediaWiki 1.26.2https://kb.ettus.com/index.php?title=Converting_an_X310_into_an_NI-USRP_Rio&diff=6028Converting an X310 into an NI-USRP Rio2024-03-26T15:46:56Z<p>MichaelDickens: note deprecation!</p>
<hr />
<div>== NOTE ==<br />
This Application Note is deprecated. Use it at your own risk!<br />
<br />
==Application Note Number==<br />
'''AN-503'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2016-05-03<br />
|style="text-align:center;"| Tim Fountain<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This Application Note explains how to use an Ettus Research-branded USRP with LabVIEW, and in effect, convert it into an NI-USRP RIO.<br />
<br />
'''NOTE:''' While this process is technically possible, NI/Ettus does ''not'' officially support it and does ''not'' guarantee providing any technical support for it. The user performs this process ''at their own risk''. The process is documented here as a convenience to users.<br />
<br />
==Summary==<br />
This document outlines the steps necessary to modify an Ettus X310 + associated daughterboards (CBX, WBX, SBX or UBX) into the identical NI model (NI USRP-294x or NI USRP-295x). Note that you must have identical daughterboards in each X310 slot for LabVIEW to function. Identical daughterboards would be 2xCBX or 2xUBX for instance.<br />
<br />
This document was created with version 15.0 of the NI USRP driver. In NI USRP 15.5 and later there will be a single LabVIEW vi that will conduct all 3 steps automatically. <br />
<br />
There are 2 steps that need to be done to make an X310 into a USRP RIO.<br />
<br />
==Step 1==<br />
The daughterboard IDs need to be burned into the EEPROM.<br />
<br />
The default location for the utilities is<br />
<br />
C:\Program Files (x86)\National Instruments\LabVIEW 2015\vi.lib\LabVIEW Targets\FPGA\USRP\niusrprio_tools.llb<br />
<br />
* Note - edit path accordingly if you have a different version of LabVIEW and/or you have installed the x64 version<br />
<br />
Use the initialize Flash.vi to load the correct daughterboard ID’s and serial numbers. The vi is auto populated with the supported daughterboard ID’s, the complete list is included in appendix 1 for reference. The serial numbers are not critical bit can be matched to the physical daughterboard serial numbers which are found on a printed label on each daughterboard.<br />
<br />
[[File:convert x310 figure1.png|500px]]<br />
<br />
The letter revision on the X310 motherboard should be to set the HW current version (rev A = 1, B =2, etc). Set the oldest compatible version to the current HW version too. For revisions 6 and below (A-F), the 3.3v device model must be set. For revisions 7 and above (G and H), use the 1.6v device model must be set.<br />
<br />
[[File:convert x310 figure2.png|500px]]<br />
<br />
==Step 2==<br />
IQ imbalance corrections need to be loaded into the EEPROM. <br />
<br />
The easiest way to load the IQ imbalance corrections into the daughterboard EEPROM is to create a sample USRP-RIO project.<br />
From the default LabVIEW windows, click create new project and select NI-USRP Simple Streaming:<br />
<br />
[[File:convert x310 figure3.png|700px]]<br />
<br />
Once the project has been created, navigate to the utilities in the project window and open the Self Correct RX IQ Offset.vi:<br />
<br />
[[File:convert x310 figure4.png|500px]]<br />
<br />
This .vi will automatically calculate the optimal IQ imbalance correction factors and load them into the factory default location on the daughterboard EEPROM. Note there is no method at this time to load DC offset corrections without sending the unit back to the factory.<br />
<br />
==Appendix 1 Motherboard and Daughterboard ID’s==<br />
<br />
===MOTHER BOARDS===<br />
{| class="wikitable"<br />
!NAME<br />
!ID<br />
|-<br />
|B200<br />
|0x7737<br />
|-<br />
|B210<br />
|0x7738<br />
|-<br />
|B200mini<br />
|0x7739<br />
|-<br />
|B205<br />
|0x773a<br />
|-<br />
|}<br />
<br />
===DAUGHTER BOARDS===<br />
====Basic and Low Frequency====<br />
{| class="wikitable"<br />
!Name (as reported by UHD)<br />
!RX ID<br />
!TX ID<br />
!Notes<br />
|-<br />
|Unknown<br />
|0xfff1<br />
|0xfff0<br />
|<br />
|- <br />
|Basic<br />
|0x0001<br />
|0x0000<br />
|<br />
|-<br />
|LF<br />
|0x000f<br />
|0x000e<br />
|<br />
|-<br />
|}<br />
<br />
====WBX====<br />
{| class="wikitable"<br />
!Name (as reported by UHD)<br />
!RX ID<br />
!TX ID<br />
!Notes<br />
|-<br />
|WBX LO<br />
|0x0051<br />
|0x0050<br />
|(Not registered in UHD)<br />
|-<br />
<br />
|WBX <br />
|0x0053<br />
|0x0052<br />
|<br />
|-<br />
<br />
|WBX + Simple GDB <br />
|0x0053<br />
|0x004f<br />
|<br />
|-<br />
<br />
|WBX v3<br />
|0x0057<br />
|0x0056<br />
|<br />
|-<br />
<br />
|WBX v3 + Simple GDB <br />
|0x0057<br />
|0x004f<br />
|<br />
|-<br />
<br />
|WBX v4 <br />
|0x0063<br />
|0x0062<br />
|<br />
|-<br />
<br />
|WBX v4 + Simple GDB <br />
|0x0063<br />
|0x004f<br />
|<br />
|-<br />
<br />
|WBX-120 <br />
|0x0081<br />
|0x0080<br />
|<br />
|-<br />
<br />
|WBX-120 + Simple GDB <br />
|0x0081<br />
|0x004f<br />
|<br />
|-<br />
|}<br />
<br />
====SBX====<br />
{| class="wikitable"<br />
!Name (as reported by UHD)<br />
!RX ID<br />
!TX ID<br />
!Notes<br />
|-<br />
|SBX<br />
|0x0054<br />
|0x0055<br />
|v3<br />
|-<br />
<br />
|SBX v4<br />
|0x0065<br />
|0x0064<br />
|<br />
|-<br />
<br />
|SBX v5<br />
|0x0069<br />
|0x0068<br />
|<br />
|-<br />
<br />
|SBX-120<br />
|0x0083<br />
|0x0082<br />
|<br />
|-<br />
|}<br />
<br />
====CBX====<br />
{| class="wikitable"<br />
!Name (as reported by UHD)<br />
!RX ID<br />
!TX ID<br />
!Notes<br />
|-<br />
|CBX<br />
|0x0067<br />
|0x0066<br />
|v3<br />
|-<br />
|CBX-120<br />
|0x0085<br />
|0x0084<br />
|<br />
|-<br />
|}<br />
<br />
====UBX====<br />
{| class="wikitable"<br />
!Name (as reported by UHD)<br />
!RX ID<br />
!TX ID<br />
!Notes<br />
|-<br />
|UBX v0.3<br />
|0x0074<br />
|0x0073<br />
|Prototype<br />
|-<br />
|UBX v0.4<br />
|0x0076<br />
|0x0075<br />
|Prototype<br />
|-<br />
|UBX-40 v1<br />
|0x0078<br />
|0x0077<br />
|<br />
|-<br />
|UBX-160 v1<br />
|0x007A<br />
|0x0079<br />
|<br />
|-<br />
|}<br />
<br />
====TwinRX====<br />
{| class="wikitable"<br />
!Name (as reported by UHD)<br />
!RX ID<br />
!TX ID<br />
!Notes<br />
|-<br />
|TwinRX v1.0<br />
|0x0091<br />
|0xffff<br />
|(0x90 Reserved)<br />
|-<br />
|}<br />
<br />
====Others (Legacy)====<br />
{| class="wikitable"<br />
!Name<br />
!ID<br />
!Notes<br />
|-<br />
|DBS Rx<br />
|0x0002<br />
|<br />
|-<br />
|TV Rx<br />
|0x0003<br />
|<br />
|-<br />
|Flex 400 Rx<br />
|0x0004<br />
|<br />
|-<br />
|Flex 900 Rx<br />
|0x0005<br />
|<br />
|-<br />
|Flex 1200 Rx<br />
|0x0006<br />
|<br />
|-<br />
|Flex 2400 Rx<br />
|0x0007<br />
|<br />
|-<br />
|Flex 400 Tx<br />
|0x0008<br />
|<br />
|-<br />
|Flex 900 Tx<br />
|0x0009<br />
|<br />
|-<br />
|Flex 1200 Tx<br />
|0x000a<br />
|<br />
|-<br />
|Flex 2400 Tx<br />
|0x000b<br />
|<br />
|-<br />
|TV Rx Rev 2<br />
|0x000c<br />
|<br />
|-<br />
|DBS Rx ClkMod<br />
|0x000d<br />
|<br />
|-<br />
|DBSRX2<br />
|0x012<br />
|<br />
|-<br />
|Flex 400 Rx MIMO A<br />
|0x0014<br />
|<br />
|-<br />
|Flex 900 Rx MIMO A<br />
|0x0015<br />
|<br />
|-<br />
|Flex 1200 Rx MIMO A<br />
|0x0016<br />
|<br />
|-<br />
|Flex 2400 Rx MIMO A<br />
|0x0017<br />
|<br />
|-<br />
|Flex 400 Tx MIMO A<br />
|0x0018<br />
|<br />
|-<br />
|Flex 900 Tx MIMO A<br />
|0x0019<br />
|<br />
|-<br />
|Flex 1200 Tx MIMO A<br />
|0x001a<br />
|<br />
|-<br />
|Flex 2400 Tx MIMO A<br />
|0x001b<br />
|<br />
|-<br />
|Flex 400 Rx MIMO B<br />
|0x0024<br />
|<br />
|-<br />
|Flex 900 Rx MIMO B<br />
|0x0025<br />
|<br />
|-<br />
|Flex 1200 Rx MIMO B<br />
|0x0026<br />
|<br />
|-<br />
|Flex 2400 Rx MIMO B<br />
|0x0027<br />
|<br />
|-<br />
|Flex 400 Tx MIMO B<br />
|0x0028<br />
|<br />
|-<br />
|Flex 900 Tx MIMO B<br />
|0x0029<br />
|<br />
|-<br />
|Flex 1200 Tx MIMO B<br />
|0x002a<br />
|<br />
|-<br />
|Flex 2400 Tx MIMO B<br />
|0x002b<br />
|<br />
|-<br />
|Flex 2200 Rx MIMO B<br />
|0x002c<br />
|<br />
|-<br />
|Flex 2200 Tx MIMO B<br />
|0x002d<br />
|<br />
|-<br />
|Flex 1800 Rx<br />
|0x0030<br />
|<br />
|-<br />
|Flex 1800 Tx<br />
|0x0031<br />
|<br />
|-<br />
|Flex 1800 Rx MIMO A<br />
|0x0032<br />
|<br />
|-<br />
|Flex 1800 Tx MIMO A<br />
|0x0033<br />
|<br />
|-<br />
|Flex 1800 Rx MIMO B<br />
|0x0034<br />
|<br />
|-<br />
|Flex 1800 Tx MIMO B<br />
|0x0035<br />
|<br />
|-<br />
|TV Rx Rev 3<br />
|0x0040<br />
|<br />
|-<br />
|DTT754<br />
|0x0041<br />
|<br />
|-<br />
|DTT768<br />
|0x0042<br />
|<br />
|-<br />
|TV Rx MIMO<br />
|0x0043<br />
|<br />
|-<br />
|TV Rx Rev 2 MIMO<br />
|0x0044<br />
|<br />
|-<br />
|TV Rx Rev 3 MIMO<br />
|0x0045<br />
|<br />
|-<br />
|TVRX2<br />
|0x0046<br />
|<br />
|-<br />
|WCDMA US<br />
|0x004d<br />
|<br />
|-<br />
|WCDMA EU<br />
|0x004e<br />
|<br />
|-<br />
|XCVR2450 Tx - No Div<br />
|0x0059<br />
|<br />
|-<br />
|XCVR2450 Tx<br />
|0x0060<br />
|<br />
|-<br />
|XCVR2450 Rx<br />
|0x0061<br />
|<br />
|-<br />
|Bitshark Rx<br />
|0x0070<br />
|<br />
|-<br />
|B150 v1 TX<br />
|0x0071<br />
|<br />
|-<br />
|B150 v1 RX<br />
|0x0072<br />
|<br />
|-<br />
<br />
<br />
|}<br />
<br />
===E3xx===<br />
{| class="wikitable"<br />
!Name<br />
!ID<br />
!Notes<br />
|-<br />
|E300 AD9364 RevB<br />
|0x0100<br />
|<br />
|-<br />
|E310 AD9361 RevB<br />
|0x0110<br />
|<br />
|-<br />
|E300 AD9364 RevC<br />
|0x0101<br />
|<br />
|-<br />
|E310 AD9361 RevC<br />
|0x0111<br />
|<br />
|-<br />
|E330<br />
|0x0120<br />
|<br />
|-<br />
|}<br />
<br />
<br />
===USB VID===<br />
{| class="wikitable"<br />
!Name<br />
!ID<br />
!Notes<br />
|-<br />
|Ettus Research<br />
|0x2500<br />
|<br />
|-<br />
|National Instruments<br />
|0x3923<br />
|<br />
|-<br />
|Cypress Semiconductor<br />
|0x04b4<br />
|<br />
|-<br />
|}<br />
<br />
===USB PID===<br />
{| class="wikitable"<br />
!Name<br />
!ID<br />
!Notes<br />
|-<br />
<br />
|FX2<br />
|0x8613<br />
|<br />
|-<br />
|FX3<br />
|0x00f3<br />
|<br />
|-<br />
|FX3 (Re-enumerated)<br />
|0x00f0<br />
|<br />
|-<br />
|B100<br />
|0x0002<br />
|<br />
|-<br />
|B200<br />
|0x0020<br />
|<br />
|-<br />
|B210<br />
|0x0020<br />
|<br />
|-<br />
|B200 (NI)<br />
|0x7813<br />
|<br />
|-<br />
|B210 (NI)<br />
|0x7814<br />
|<br />
|-<br />
|B200mini<br />
|0x0021<br />
|<br />
|-<br />
|B205<br />
|0x0022<br />
|<br />
|-<br />
|}<br />
<br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Application_Notes&diff=6027Application Notes2024-03-26T15:43:52Z<p>MichaelDickens: remove RIO convert AN ... is still searchable</p>
<hr />
<div>Application Notes (AN) and technical articles written by engineers, for engineers. These articles offer experienced analysis, design ideas, reference designs, and tutorials—to make you productive and successful using USRP devices.<br />
<br />
{| class="wikitable"<br />
!colspan="4"|Application Notes<br />
|-<br />
<br />
<br />
! style="text-align:center;"| Number<br />
! style="text-align:center;"| Title<br />
! style="text-align:center;"| Abstract<br />
! style="text-align:center;"| Author(s)<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-009<br />
|style="width: 30%;"| [[Declaration of Conformity]]<br />
|style="width: 50%;"| This application note describes how to find Declaration of Conformity information for a given Ettus device. <br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-010<br />
|style="width: 30%;"| [[Trade Compliance and Export Control Classification Number (ECCN)]]<br />
|style="width: 50%;"| This application note describes how to find trade compliance information including the ECCN for a given Ettus device. <br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-055<br />
|style="width: 30%;"| [[About Sampling Rates and Master Clock Rates for the USRP X440]]<br />
|style="width: 50%;"| This application note guides users through the selection process of Master Clock Rates (MCR) for the USRP X440.<br />
|style="width: 10%; text-align: center;"| Marian Koop<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-088<br />
|style="width: 30%;"| [[USRP Host Performance Tuning Tips and Tricks]]<br />
|style="width: 50%;"| This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-111<br />
|style="width: 30%;"| [[UHD Device Eraser and Certificates of Volatility]]<br />
|style="width: 50%;"| This AN provides an overview of the UHD Device Eraser utility as well as links to the Certificates of Volatility for all Ettus products.<br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-117<br />
|style="width: 30%;"| [[GPSDO Selection Guide]]<br />
|style="width: 50%;"| This AN explains how to select and use a GPSDO with the USRP B-, N-, and X-series devices.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-121<br />
|style="width: 30%;"| [[Debugging FPGA images]]<br />
|style="width: 50%;"| This application note covers the basics to get you through the process of probing the signals inside an FPGA. In order to accomplish that, we will review briefly the 'Xilinx ChipScope Analyzer' and will apply it to one of our core RFNoC blocks: the RFNoC Signal generator. <br />
|style="width: 10%; text-align: center;"| Nicolas Cuervo <br> Sugandha Gupta <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-142<br />
|style="width: 30%;"| [[Transmitting DVB-S2 with GNU Radio and an USRP B210]]<br />
|style="width: 50%;"| This application note will demonstrate using an USRP B210 and the GNU Radio DTV example flowgraph to transmit a DVB-S2 video stream to an off-the-shelf satellite receiver. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-158<br />
|style="width: 30%;"| [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]]<br />
|style="width: 50%;"| This application note provides instructions for synchronizing multiple USRP N3xx devices using White Rabbit Ethernet-based synchronization. <br />
|style="width: 10%; text-align: center;"| Dan Baker<br />
Wan Liu <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-177<br />
|style="width: 30%;"| [[About USRP Bandwidths and Sampling Rates]]<br />
|style="width: 50%;"| This AN provides insight into the topics of USRP architecture, system bandwidth, host interface throughput, and available sampling rates.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-178<br />
|style="width: 30%;"| [[Resolving Audio Codec Enumeration Issues On The E31x]]<br />
|style="width: 50%;"| This application note covers Resolving Audio Codec Enumeration Issues On The E31x. <br />
|style="width: 10%; text-align: center;"| Logan Fagg<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-188<br />
|style="width: 30%;"| [[Interrogating Passive Wireless SAW Sensors with the USRP]]<br />
|style="width: 50%;"| Typical interrogator design for wireless SAW sensor systems require many discrete components and lengthy build times, making it difficult to rapidly adapt to sensor designs in a research environment. We have employed the USRP B200 as a SAW sensor interrogation system. Interrogation of wideband orthogonal frequency coded (OFC) SAW sensors imposes strict requirements on the timing and synchronization of the transceiver. The USRP FPGA has been modified to operate in a synchronous, pulsed mode of operation, allowing rapid data acquisition and the full 56MHz bandwidth to be utilized. Data from the USRP is passed to a custom matched filter correlator routine to extract sensor parameters. The system is capable of interrogating multiple sensors, simultaneously. Demonstration of the system is accomplished by wirelessly interrogating SAW sensors at 915MHz and extracting temperature.<br />
|style="width: 10%; text-align: center;"| Trip Humphries<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-204<br />
|style="width: 30%;"| [[Getting Started with UHD and C++]]<br />
|style="width: 50%;"| This AN explains how to write and build C++ programs that use the UHD API and introduces<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-244<br />
|style="width: 30%;"| [[Direction Finding with the USRP™ X-Series and TwinRX™]]<br />
|style="width: 50%;"| This application note covers using the USRP™ TwinRX™ daughterboard in a direction find application using the MUSIC algorithm. <br />
|style="width: 10%; text-align: center;"| Srikanth Pagadarai <br> Travis Collins <br> Alexander M. Wyglinski<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-296<br />
|style="width: 30%;"| [[Using Dual 10 Gigabit Ethernet on the USRP X300/X310]]<br />
|style="width: 50%;"| This short guide is meant to help in quickly setting up an X-series USRP for use over two 10 Gigabit Ethernet links simultaneously. <br />
|style="width: 10%; text-align: center;"| Paul David<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-305<br />
|style="width: 30%;"| [[X300/X310 Device Recovery]]<br />
|style="width: 50%;"| This application note covers the details of recovering the USRP X300/X310 via JTAG. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-309<br />
|style="width: 30%;"| [[About the Motherboard and Daughtercard EEPROM on USRP Devices]]<br />
|style="width: 50%;"| This AN discusses the EEPROM storage on various USRP devices and daughtercards. This guides explains how to update the EEPROM contents and recover from EEPROM corruption. The product codes, which are also stored in the EEPROM, for all USRP devices and daughtercards are also given for reference.<br />
|style="width: 10%; text-align: center;"| Trip Humphries<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-311<br />
|style="width: 30%;"| [[Software Development on the E310 and E312]]<br />
|style="width: 50%;"| This application note covers the software development process on the USRP E310 and E312. <br />
|style="width: 10%; text-align: center;"| Martin Braun <br> Nicolas Cuervo<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-315<br />
|style="width: 30%;"| [[Software Development on the E3xx USRP - Building RFNoC UHD / GNU Radio / gr-ettus from Source]]<br />
|style="width: 50%;"| This application note is one of a multi-part series which will cover the software development process on the USRP E310, E312 and E313. It will cover building the rfnoc-devel branch of UHD, GNU Radio and gr-ettus from source for the host machine, and cross-compiling the rfnoc-devel branch of UHD, GNU Radio and gr-ettus for the E3xx USRP. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-322<br />
|style="width: 30%;"| [[Experiments with the UBX Daughterboard in the HF Band]]<br />
|style="width: 50%;"| We show the results of experiments with the UBX daughtercard on an USRP X310 platform for use in the HF frequency range, from 1.8MHz to 30MHz. While the UBX is nominally rated for use only down to 10 MHz, with careful flow-graph design, and pre-filtering, it provides quite-good performance across the HF bands.<br />
|style="width: 10%; text-align: center;"| Marcus Leech<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-325<br />
|style="width: 30%;"| [[N200/N210 Device Recovery]]<br />
|style="width: 50%;"| This application note covers the details of recovering your N200/N210.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-335<br />
|style="width: 30%;"| [[Streaming processed data from the E31x with GNU Radio and ZMQ]]<br />
|style="width: 50%;"| This application note will demonstrate using the USRP E310 to remotely stream processed data to a host machine. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-355<br />
|style="width: 30%;"| [[Modifying an X310 Chassis for External LO Sharing]]<br />
|style="width: 50%;"| This document describes how to modify an X310 chassis to wire the LO out of the back plate. Doing this will allow the user to export and import an LO signal as desired when using a compatible daughterboard such as the TwinRX. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-363<br />
|style="width: 30%;"| [[Implementation of an ADS-B/Mode-S Receiver in GNU Radio]]<br />
|style="width: 50%;"| This AN guides the reader through the implementation of an ADS-B receiver using the gr-air-modes Out-of-Tree (OOT) module for GNU Radio. An explanation of ADS-B is also provided, and several real-world, over-the-air examples and profiled.<br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-400<br />
|style="width: 30%;"| [[Getting Started with RFNoC in UHD 4.0]]<br />
|style="width: 50%;"| This AN describes how use RFNoC in UHD 4.0, including building FPGA images for RFNoC, changing which blocks are included in the build, and creating your own RFNoC blocks.<br />
|style="width: 10%; text-align: center;"| Sugandha Gupta <br> Brent Stapleton <br> Wade Fife <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-401<br />
|style="width: 30%;"| [[RFNoC 4 Migration Guide]]<br />
|style="width: 50%;"| Guide on how to migrate RFNoC blocks written for RFNoC 3 to RFNoC 4.<br />
|style="width: 10%; text-align: center;"| Jonathon Pendlum<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-444<br />
|style="width: 30%;"| [[Using B200/B210/B200mini/B205mini on OSX / macOS with UHD]]<br />
|style="width: 50%;"| This AN provides a basic guide for what to expect when using a USB-based B-series USRP on OSX / macOS with UHD.<br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-445<br />
|style="width: 30%;"| [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Linux platform. Other alternate installation methods are also discussed.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-452<br />
|style="width: 30%;"| [[5G NR EVM Measurements with the USRP N320/N321]]<br />
|style="width: 50%;"| Example EVM measurements are shown using the USRP N320/N321 receiver and the 5G New Radio (5G NR) modulation standard. The use of I/Q image calibration and spur-dodging are demonstrated as methods to improve EVM performance. <br />
|style="width: 10%; text-align: center;"| Drew Fischer<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-492<br />
|style="width: 30%;"| [[Selecting a RF Daughterboard]]<br />
|style="width: 50%;"| This AN explores the RF daughterboards used by the N-series and X-series USRP devices at a high level, compares devices across several primary features, and walks the reader through the process of selecting a particular device for the their application.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-500<br />
|style="width: 30%;"| [[Getting Started with DPDK and UHD]]<br />
|style="width: 50%;"| This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
|style="width: 10%; text-align: center;"| Nate Temple <br> Alex Williams <br> Wade Fife <br> Matt Prost<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-504<br />
|style="width: 30%;"| [[USRP N Series Quick Start (Daughterboard Installation)]]<br />
|style="width: 50%;"| This application note is a detailed step-by-step guide to install a daughterboard into the USRP N200/N210.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-524<br />
|style="width: 30%;"| [[Building and Installing UHD and GNU Radio in an Offline Environment]]<br />
|style="width: 50%;"| This application note will provide step-by-step instructions on building and installing UHD and GNU Radio in an offline environment. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-525<br />
|style="width: 30%;"| [[Building and Installing UHD and GNU Radio to a Custom Prefix]]<br />
|style="width: 50%;"| This application note provides step-by-step instructions on building and installing UHD and GNU Radio to a local directory. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-561<br />
|style="width: 30%;"| [[Implementation of a Simple FM Receiver in GNU Radio]]<br />
|style="width: 50%;"| This AN shows a quick and simple implementation of an FM receiver for the USRP using GNU Radio. The goal is to easily demonstrate a practical application, and to verify that the USRP is functioning properly.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-611<br />
|style="width: 30%;"| [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Windows platform.<br />
|style="width: 10%; text-align: center;"| Derek Kozel<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-620<br />
|style="width: 30%;"| [[Troubleshooting X300/X310 Device Discovery Issues]]<br />
|style="width: 50%;"| Troubleshooting guide to intended to cover some of the most commonly recommended steps to enable USRP connectivity. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-621<br />
|style="width: 30%;"| [[Troubleshooting N310/N320 Device Discovery Issues]]<br />
|style="width: 50%;"| Troubleshooting guide to intended to cover some of the most commonly recommended steps to enable USRP connectivity. Serves as a supplement to the N3xx getting started guide. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-630<br />
|style="width: 30%;"| [[Writing the USRP File System Disk Image to a SD Card]]<br />
|style="width: 50%;"| This application note will provide step-by-step instructions on writing a file system disk image to a SD card using Linux. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-638<br />
|style="width: 30%;"| [[Running UHD and GNU Radio on NI USRP-RIO]]<br />
|style="width: 50%;"| This AN explains the process to updating your NI USRP-RIO to run UHD and GNU Radio. <br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple <br> Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-642<br />
|style="width: 30%;"| [[Using the RFNoC Replay Block]]<br />
|style="width: 50%;"| This application note guides a user through basic use of the RFNoC Replay block in UHD 3.x and explains how to run the UHD Replay example. This example covers use on the X300/X310 and N310 products. <br />
|style="width: 10%; text-align: center;"| Wade Fife<br />
|-<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-642b<br />
|style="width: 30%;"| [[Using the RFNoC Replay Block in UHD 4]]<br />
|style="width: 50%;"| This application note guides a user through basic use of the RFNoC Replay block in UHD 4.x and explains how to run the UHD Replay example. <br />
|style="width: 10%; text-align: center;"| Martin Braun<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-666<br />
|style="width: 30%;"| [[Mean Time Between Failure (MTBF) of USRPs and Daughterboards]]<br />
|style="width: 50%;"| This AN provides information about the MTBF for USRPs and daughterboards<br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-725<br />
|style="width: 30%;"| [[USRP N320/N321 LO Distribution]]<br />
|style="width: 50%;"| This application note provides an overview of using the LO Distribution of the N320/N321 USRPs.<br />
|style="width: 10%; text-align: center;"| Brian Avenell <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-732<br />
|style="width: 30%;"| [[USRP E312 Battery Replacement Instructions]]<br />
|style="width: 50%;"| This application note covers replacing the battery cell inside the USRP E312. <br />
|style="width: 10%; text-align: center;"| Robin Coxe<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-788<br />
|style="width: 30%;"| [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Mac OS X platform.<br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-800<br />
|style="width: 30%;"| [[Enabling Ethernet Connectivity on Octoclock and Octoclock-G]]<br />
|style="width: 50%;"| This document supplements the UHD Manual's guide for updating the Octoclock bootloader to allow for Ethernet communications with the device. <br />
|style="width: 10%; text-align: center;"| Sam Reiter <br> Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-822<br />
|style="width: 30%;"| [[Multichannel RF Reference Architecture]]<br />
|style="width: 50%;"| This application note provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture. <br />
|style="width: 10%; text-align: center;"| Michael Dickens <br> Neel Pandeya <br> Jovian Wysocki<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-823<br />
|style="width: 30%;"| [[Getting Started with RFNoC Development]]<br />
|style="width: 50%;"| This application note gives a brief introduction into the steps required to start developing RFNoC blocks on your computer with UHD 3. <br />
|style="width: 10%; text-align: center;"| Martin Braun <br> Nicolas Cuervo<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-832<br />
|style="width: 30%;"| [[Mapping Between ER-USRP and NI-USRP Product Numbers]]<br />
|style="width: 50%;"| This application note covers the details of the mapping between Ettus Research USRP and National Instruments USRP product numbers. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-881<br />
|style="width: 30%;"| [[Selecting a USRP Device]]<br />
|style="width: 50%;"| This AN explores the USRP family at a high level, compares devices across several primary features, and walks the reader through the process of selecting a particular device for the their application.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-882<br />
|style="width: 30%;"| [[Synchronization and MIMO Capability with USRP Devices]]<br />
|style="width: 50%;"| Discusses the requirements for Multiple-In-Multiple-Out (MIMO) and phased-array systems. Summarizes the MIMO capability of each USRP device and daughterboard, and shows how to build MIMO systems with the USRP product family.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-883<br />
|style="width: 30%;"| [[Synchronizing USRP Events Using Timed Commands in UHD]]<br />
|style="width: 50%;"| Guide to cover common USRP synchronization scenarios and deep-dive into the use of timed commands within USRPs. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-888<br />
|style="width: 30%;"| [[Getting Started with 4G LTE using Eurecom OpenAirInterface (OAI) on the USRP 2974]]<br />
|style="width: 50%;"| Discusses how to install and configure the OpenAirInterface (OAI) software on the USRP 2974 hardware to implement a 4G LTE cellular basestation (eNodeB).<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-904<br />
|style="width: 30%;"| [[USRP X Series Quick Start (Daughterboard Installation)]]<br />
|style="width: 50%;"| This application note is a detailed step-by-step guide to install a daughterboard into the USRP X300/X310.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-936<br />
|style="width: 30%;"| [[Verifying the Operation of the USRP Using UHD and GNU Radio]]<br />
|style="width: 50%;"| This AN explains how to use UHD and GNU Radio, once installed, to verify the correct operation of the USRP. Several test procedures are explained in detail. Several tests make use of an optional spectrum analyzer and signal generator.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-956<br />
|style="width: 30%;"| [[OAI Reference Architecture for 5G and 6G Research with USRP]]<br />
|style="width: 50%;"| This application note discusses how to build and implement 5G networks using USRP radios and the Eurecom OAI software stack.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Bharat Agarwal <br> Gerardo Trevino<br />
|-<br />
<br />
|}</div>MichaelDickenshttps://kb.ettus.com/index.php?title=E320_Getting_Started_Guide&diff=6026E320 Getting Started Guide2024-03-20T14:04:27Z<p>MichaelDickens: /* Autoboot */ update link to firmware archive & extracted directory name</p>
<hr />
<div>==Kit Contents==<br />
<br />
===E320 Board-only===<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP E320<br />
* Power connector (assembly required) <br />
* 4 M3x0.5, M3x5 Standoffs <br />
* 1 Gb Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* 1 Gb SFP+ to RJ45 Adapter<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
|[[File:e320 board only.jpg|500px|center]]<br />
|}<br />
<br />
===E320 Full Enclosure===<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP E320 in enclosure <br />
* DC Power Supply (12V, 7A)<br />
* 1 Gb Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* 1 Gb SFP+ to RJ45 Adapter<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
* T8 Torx Wrench<br />
|[[File:e320 enclosure kit.jpg|500px|center]]<br />
|}<br />
<br />
==Verify the Contents of Your Kit==<br />
Ensure that your kit contains all the items listed above. If any items are missing, please contact sales@ettus.com immediately.<br />
<br />
==You Will Need==<br />
<br />
* For Network Mode: A host computer with an 1 or 10 Gb Ethernet interface. If operating with the 10 Gb Ethernet interface, the "XG" FPGA image must be loaded before the SFP+ port will operate at 10 Gb speeds. Optionally a second 1 Gb Ethernet interface can be used to connect to the onboard ARM CPU for remote management. <br />
<br />
* For Embedded Mode: A host computer is only required for initial device configuration, remote control and management, or data visualization. The host computer can connect to the RJ45 1 Gb port or Serial Console port to remotely access the Open Embedded Linux operating system running on the ARM CPU. Once configured, the USRP E320 can operate as a stand-alone device without a connection to a remote host computer. <br />
<br />
* For Board-only Version: A third-party 10-14V/3A power supply, which requires assembly with the power connect components included in the kit. An assembled power supply can be purchased here: https://www.ettus.com/product/details/12V-PWR<br />
<br />
==Proper Care and Handling==<br />
All Ettus Research products are individually tested before shipment. The USRP is guaranteed to be functional at the time it is received by the customer. Improper use or handling of the USRP can cause the device to become non-functional. Take the following precautions to prevent damage to the unit.<br />
<br />
* Never allow anything especially metal objects to touch the board while it is powered on. <br />
* Always properly terminate the transmit port with an antenna or 50Ω load.<br />
* Always handle the board with proper anti-static methods.<br />
* Never allow the board to directly or indirectly come into contact with any voltage spikes.<br />
* Never allow any water or condensing moisture to come into contact with the device.<br />
* Always use caution with FPGA, firmware, or software modifications.<br />
* Never touch the circuit board or heatsink while the device is powered on. <br />
* All connections should be made/removed while is device is powered off. <br />
<br />
{|<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |Never apply more than -15 dBm of power into any RF input.<br />
|-<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |Always use at least 30dB attenuation if operating in loopback configuration<br />
|-<br />
|}<br />
<br />
==Install and Setup the Software Tools on Your Host Computer==<br />
<br />
To use your Universal Software Radio Peripheral (USRP™), you must have software tools correctly installed and configured on your host computer. Step-by-step guides for these software tools are found in the Application Notes for Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux|Linux]], [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X|OS X]] and [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows|Windows]].<br />
<br />
The USRP E320 requires UHD version 3.13.0.2 or later. It is strongly recommended to use the latest stable release of UHD on both the host computer and the USRP via the filesystem on the SD card. If this release fails to work in some way, then try the maintenance branch of the latest stable version. If you are operating the device in Network Mode, the version of UHD running on the host machine and E320 USRP must match to within the same maintenance release and branch. See the [https://github.com/ettusresearch/uhd UHD GitHub repository] for the latest release and maintenance branch.<br />
<br />
==Connecting the Device==<br />
===Interfaces Overview===<br />
Listed below are the interfaces to connect to the USRP E320. Each interface has specific functionality, limitations and purpose.<br />
<br />
'''Serial Console'''<br />
<br />
The Serial Console provides a low-level interface to the ARM CPU and STM32 microcontroller, typically used for debugging. The serial console can also be used as a JTAG connection to the FPGA.<br />
<br />
'''1 Gb RJ45 Connection'''<br />
<br />
The 1 Gb RJ45 Connection interfaces with the on-board ARM CPU. When operated in "Network mode", this interface can optionally be used for remote control and management traffic. Regardless of the operation mode (Host vs Embedded) this interface can be used to connect to the ARM via SSH. By default, the 1 Gb RJ45 connection is configured to use a DHCP assigned IP address.<br />
<br />
'''SFP+ Connection'''<br />
<br />
The SFP+ Connection supports multiple interfaces for streaming high-speed, low-latency data, depending upon which FPGA image is loaded.<br />
<br />
===Setting up a Serial Console Connection===<br />
It is possible to gain shell access to the device using a serial terminal emulator via the Serial Console port. Most Linux, OS X, or other Unix based operating systems have a utility called <code>screen</code> which can be used for this purpose. <br />
<br />
If you do not have <code>screen</code> installed, it can be installed via your distribution's package manager. For Ubuntu/Debian based operating systems it can be installed with the package manager <code>apt</code> such as:<br />
<br />
sudo apt install screen<br />
<br />
The default Baud Rate for the Serial Console is: <code>115200</code><br />
<br />
The exact device node you should attach to depends on your operating system's driver and other USB devices that might already be connected. Modern Linux systems offer alternatives to simply trying device nodes; instead, the OS might have a directory of symlinks under <code>/dev/serial/by-id</code>:<br />
<br />
$ ls /dev/serial/by-id<br />
usb-FTDI_Dual_RS232-HS-if00-port0<br />
usb-FTDI_Dual_RS232-HS-if01-port0<br />
usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6A69-if00-port0<br />
usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6A69-if01-port0<br />
<br />
NOTE: Exact names depend on the host operating system version and may differ.<br />
<br />
Every E320 series device connected to USB will by default show up as four different devices. The devices labeled <code>"USB_to_UART_Bridge_Controller"</code> are the devices that offer a serial prompt. The first (with the <code>if00</code> suffix) connects to the <code>STM32 Microcontroller</code>, whereas the second connects to the <code>ARM CPU</code>.<br />
<br />
If you have multiple E320 Serial Consoles connected to a single host, you may have to empirically test nodes.<br />
<br />
Connecting to the ARM CPU can be performed with the command:<br />
<br />
$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6A69-if01-port0 115200<br />
<br />
Upon starting the USRP E320, boot messages will appear and rapidly update. Once the boot process successfully completes, a login prompt like the following should appear:<br />
<br />
Alchemy 2018.04 ni-e320-serial ttyPS0<br />
ni-e320-serial login:<br />
<br />
Enter the username: <code>root</code><br />
<br />
By default, the <code>root</code> user's password is left blank. Press the <code>Enter</code> key when prompted for a password.<br />
<br />
You should now be presented with a shell prompt similar to the following:<br />
<br />
root@ni-e320-<motherboard serial #>:~#<br />
<br />
Using the default configuration, the serial console will show all kernel log messages (which are not available when using SSH) and give access to the boot loader (U-boot prompt). This can be used to debug kernel or boot-loader issues more efficiently than when logged in via SSH.<br />
<br />
===Connecting to the microcontroller===<br />
<br />
Using the Serial Console interface, it is possible to connect to the STM32 microcontroller with the command below. The STM32 controls the power sequencing and several other low-level device operations.<br />
<br />
$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6A69-if00-port0 115200<br />
<br />
The STM32 interface provides a very simple prompt. The command <code>help</code> will list all available commands. A direct connection to the microcontroller can be used to hard-reset the device without physically accessing it (i.e., emulating a power button press) and other low-level diagnostics.<br />
<br />
===Connecting to the ARM via SSH===<br />
By default, the RJ45 1 Gb management interface is configured to be assigned a DHCP IP address.<br />
<br />
If you have access to a network which provides a DHCP server (such as a common router's LAN), attach the RJ45 1 Gb port to this network. Details vary by vendor, however, most router management interfaces will provide a list of attached devices to the LAN including their IP address.<br />
<br />
Without access to a router management interface, you can identify the IP address by connecting to the ARM CPU via Serial Console as detailed in the section above and running the command <code>ip a</code>:<br />
<br />
Example Output:<br />
<br />
<pre><br />
# ip a<br />
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000<br />
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br />
inet 127.0.0.1/8 scope host lo<br />
valid_lft forever preferred_lft forever<br />
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
inet 192.168.1.151/24 brd 192.168.1.255 scope global dynamic eth0<br />
valid_lft 42865sec preferred_lft 42865sec<br />
3: sfp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8000 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
inet 192.168.10.2/24 brd 192.168.10.255 scope global sfp0<br />
valid_lft forever preferred_lft forever<br />
</pre><br />
<br />
If you do not have access to a network with a DHCP server, you can create one using the Linux utility <code>dnsmasq</code>:<br />
<br />
$ sudo dnsmasq -i <ETHERNET_ADAPTER_NAME> --dhcp-range=192.168.1.50,192.168.1.100 --except-interface=lo --bind-dynamic --no-daemon<br />
<br />
NOTE: Modify the value <code><ETHERNET_ADAPTER_NAME></code> to match the interface you would like to create a DHCP server on.<br />
<br />
After the device has obtained an IP address, you can remotely log into it from a Linux or macOS systems with SSH, as shown below:<br />
<br />
$ ssh root@192.168.1.51<br />
<br />
NOTE: The IP address may vary depending on your network setup.<br />
<br />
NOTE: The <code>root</code> password is empty/blank.<br />
<br />
On Microsoft Windows, the SSH connection can be established using the third-party program, such as PuTTY.<br />
<br />
After logging in, you should be presented with a shell prompt like the following:<br />
<br />
root@ni-e320-<motherboard serial #>:~#<br />
<br />
==Updating the Linux File System==<br />
Before operating the device, it is strongly recommended to update to the latest version of the Embedded Linux file system. If you are operating the device in Network Mode, the version of UHD running on the host machine and E320 USRP must match. <br />
<br />
There is two ways to update the file system for the E320 USRP: <br />
<br />
1. Mender<br />
<br />
2. Physically remove microSD card from device and write a new file system to the microSD card. <br />
<br />
===File System Partition Layout===<br />
The SD Card is divided into four partitions. There are two root file system partitions, a "boot" partition and a "data" partition. <br />
<br />
Any data you would like to preserve through Mender updates should be saved to the "data" partition, which is mounted at <code>/data</code>.<br />
<br />
===Updating the file system with Mender===<br />
Mender is third-party software that enables remote updating of the root file system without physically accessing the device (see also the Mender website https://mender.io). Mender can be executed locally on the device, or a Mender server can be set up which can be used to remotely update an arbitrary number of USRP devices. Users can host their own local Mender server, or use servers hosted by Mender as a paid service; contact Mender for more information. <br />
<br />
====Mender Update Process====<br />
When updating the file system using Mender, the tool will overwrite the root file system partition that is not currently mounted. Any data stored in the root partitions will be permanently lost with a Mender update.<br />
<br />
After updating a partition with Mender, it will reboot into the newly updated partition. Only if the update is confirmed by the user, the update will be made permanent. This means that if an update fails, the device will be always able to reboot into the partition from which the update was originally launched, which presumably is in a working state. Another update can be launched now to correct the previous, failed update, until it works.<br />
<br />
To obtain the file system Mender image (these are files with a <code>.mender</code> suffix), run the following command on the host computer with Internet access:<br />
<br />
$ sudo uhd_images_downloader -t mender -t e320 --yes<br />
<br />
Example Output:<br />
[INFO] Using base URL: https://files.ettus.com/binaries/cache/<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
[INFO] No inventory file found at /usr/local/share/uhd/images/inventory.json. Creating an empty one.<br />
301483 kB / 301483 kB (100%) e3xx_e320_mender_default-v4.4.0.0.zip<br />
[INFO] Images download complete.<br />
<br />
NOTE: In the output of the command, the folder destination where the images are saved is printed out.<br />
<br />
Next, you will need to copy this Mender file system image to the USRP E320. This can be done with the Linux utility <code>scp</code>.<br />
<br />
$ scp /usr/local/share/uhd/images/usrp_e320_fs.mender root@192.168.1.51:~/. <br />
<br />
Note: The path and IP may different for your configuration, the command above assumes you're using the default installation path of <code>/usr/local</code> and that the E320's IP is <code>192.168.1.51</code>.<br />
<br />
After copying the Mender file system image to the E320, connect to the E320 using either the Serial Console, or via SSH to gain shell access.<br />
<br />
On the E320, run <code>mender install /path/to/latest.mender</code> to update the file system:<br />
<br />
root@ni-e320-serial:~# mender install /home/root/usrp_e320_fs.mender<br />
<br />
Example Output:<br />
<br />
<pre><br />
root@ni-e320-316E375:~# mender install /home/root/usrp_e320_fs.mender <br />
INFO[0000] Start updating from local image file: [/home/root/usrp_e320_fs.mender] module=rootfs<br />
Installing update from the artifact of size 399640064<br />
INFO[0000] opening device /dev/mmcblk0p3 for writing module=block_device<br />
INFO[0000] partition /dev/mmcblk0p3 size: 2046820352 module=block_device<br />
................................ 0% 1024 KiB<br />
................................ 0% 2048 KiB<br />
................................ 0% 3072 KiB<br />
[truncated for readability]<br />
................................ 99% 389120 KiB<br />
................................ 99% 390144 KiB<br />
................................ 100% 390273 KiB<br />
INFO[0740] wrote 2046820352/2046820352 bytes of update to device /dev/mmcblk0p3 module=device<br />
INFO[0744] Enabling partition with new image installed to be a boot candidate: 3 module=device<br />
<br />
</pre><br />
<br />
The artifact can also be stored on a remote server:<br />
$ mender install <http://server.name/path/to/latest.mender><br />
<br />
This procedure will take a few minutes to complete. After mender has logged a successful update, reboot the device:<br />
$ reboot<br />
<br />
If the reboot worked, and the device seems functional, commit the changes so that the boot loader knows to permanently boot into this partition:<br />
$ mender -commit<br />
<br />
To identify the currently installed Mender artifact from the command line, the following file can be queried on the E320:<br />
$ cat /etc/mender/artifact_info<br />
<br />
If you are using a Mender server, the updates can be initiated from a web dashboard. From there, you can start the updates without having to log into the device, and you can update groups of USRPs with a few clicks in a web GUI. The dashboard can also be used to inspect the state of USRPs. This is a simple way to update groups of rack-mounted USRPs with custom file systems.<br />
<br />
For more information on updating the file-system, refer to the E3xx page in the Devices section of the UHD Manual at http://uhd.ettus.com.<br />
<br />
===Updating the files system by writing the disk image===<br />
The microSD card is accessible directly on the Board-only version of the E320 USRP. The E320 Full Enclosure version must be opened with the included Torx wrench. <br />
<br />
NOTE: This method will overwrite all data saved on the microSD card, including any data saved to the <code>/data</code> partition.<br />
<br />
Please see the separate application note, [[Writing the USRP File System Disk Image to a SD Card]], for step-by-step instructions on writing the file system image to the microSD card.<br />
<br />
==Updating the Network Configurations==<br />
The USRP E320 systemd network configuration files are located either at: <code>/etc/systemd/network/</code><br />
<br />
# ls /etc/systemd/network/<br />
eth0.network sfp0.network <br />
<br />
or for newer versions of the file system: <code>/data/network/</code><br />
<br />
# ls /data/network/<br />
eth0.network int0.network sfp0.network<br />
<br />
For details on configuration please refer to the [https://www.freedesktop.org/software/systemd/man/systemd.network.html systemd-networkd manual pages].<br />
<br />
The factory settings are as follows:<br />
<pre><br />
eth0 (DHCP):<br />
<br />
[Match]<br />
Name=eth0<br />
<br />
[Network]<br />
DHCP=v4<br />
<br />
[DHCPv4]<br />
UseHostname=false<br />
<br />
sfp0 (static):<br />
<br />
[Match]<br />
Name=sfp0<br />
<br />
[Network]<br />
Address=192.168.10.2/24<br />
<br />
[Link]<br />
MTUBytes=8000<br />
</pre><br />
<br />
Additional notes on networking:<br />
<br />
* Care needs to be taken when editing these files on the device, since <code>vi</code> / <code>vim</code> sometimes generates undo files (e.g. <code>/data/network/sfp0.network~</code>), that <code>systemd-networkd</code> might accidentally pick up.<br />
* Temporarily setting the IP addresses or MTU sizes via <code>ifconfig</code> or other command line tools will only change the value until the next reboot or reload of the FPGA image.<br />
* If the MTU of the device and host computers differ, streaming issues can occur.<br />
* Streaming via SFP0 at 1 Gb rates requires a MTU of <code>1500</code><br />
* Streaming via SFP0 at 10 Gb rates requires a MTU of <code>8000</code><br />
<br />
For addition details on network configuration here: https://files.ettus.com/manual/page_usrp_e320.html#e320_network_configuration<br />
<br />
==Updating the FPGA Image==<br />
<br />
===Network mode FPGA Image Update===<br />
The FPGA image should match the version of UHD installed on the host computer when operated in Network mode. <br />
<br />
Network mode FPGA image updates must be made through the RJ45 management interface.<br />
<br />
To obtain all the FPGA images for your installed version of UHD, run the following command on the host computer with internet access:<br />
<br />
$ sudo uhd_images_downloader -t e320 -t fpga<br />
<br />
Example Output:<br />
<br />
$ uhd_images_downloader -t e320 -t fpga<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
[INFO] No inventory file found at /usr/local/share/uhd/images/inventory.json. Creating an empty one.<br />
05920 kB / 05920 kB (100%) e3xx_e320_fpga_default-g494ae8bb.zip<br />
[INFO] Images download complete.<br />
<br />
There is two versions of the E320 FPGA images shipped with UHD:<br />
<br />
- <code>1G</code> for 1 Gb rates on the SFP+ port (default image)<br />
<br />
- <code>XG</code> for 10 Gb rates on the SFP+ port<br />
<br />
In this example, we load the <code>XG</code> variant of the FPGA image.<br />
<br />
$ uhd_image_loader --args "type=e3xx,mgmt_addr=<E320_RJ45_IP_ADDR>,fpga=XG"<br />
<br />
Example Output:<br />
<br />
$ uhd_image_loader --args "mgmt_addr=192.168.1.51,type=e3xx,fpga=XG"<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.13.1.0-1-gd3b7e90a<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.51,type=e3xx,product=e320,serial=316E375,claimed=False,skip_init=1<br />
[INFO] [MPMD] Claimed device without full initialization.<br />
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.<br />
[INFO] [MPM.PeriphManager] Updating component `fpga'<br />
[INFO] [MPM.PeriphManager] Updating component `dts'<br />
[INFO] [MPM.RPCServer] Resetting peripheral manager.<br />
[INFO] [MPM.PeriphManager] Device serial number: 316E375<br />
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.<br />
[INFO] [MPM.PeriphManager] Found 1 daughterboard(s).<br />
<br />
The FPGA is immediately updated, and this FPGA image will continue to be used. The device does not need to be power cycled to use the new image. <br />
<br />
To load a different FPGA image (i.e. <code>1G</code>), modify the device argument <code>fpga=</code> to a value of <code>fpga=1G</code>.<br />
<br />
To specify the path to a custom FPGA image, use the <code>--fpga-path</code> argument.<br />
<br />
$ uhd_image_loader --args "type=e3xx,mgmt_addr=<E320_RJ45_IP_ADDR>" --fpga-path=/path/to/custom/fpga.bit<br />
<br />
<br />
{|<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |The Verilog code for the FPGA in the USRP E320 is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
|-<br />
|}<br />
<br />
<br />
===Embedded Mode FPGA Image Update===<br />
<br />
It is possible to update the FPGA image when operated in Embedded mode. Connect to the ARM CPU [[#Setting_up_a_Serial_Console_Connection|via Serial Console]] or [[E320_Getting_Started_Guide#Connecting_to_the_ARM_via_SSH| via SSH]]. It is generally recommend to use SSH over the RJ45 interface for remote management. <br />
<br />
Run the command <code>uhd_images_downloader</code> to download the FPGA images to the device's file system:<br />
<br />
NOTE: The 1 Gb RJ45 management interface will require Internet access for this next step.<br />
<br />
<pre><br />
root@ni-e320-serial:~# python3 /usr/bin/uhd_images_downloader -t e320 -t fpga<br />
[INFO] Images destination: /usr/share/uhd/images<br />
[INFO] No inventory file found at /usr/share/uhd/images/inventory.json. Creating an empty one.<br />
05920 kB / 05920 kB (100%) e3xx_e320_fpga_default-g494ae8bb.zip<br />
[INFO] Images download complete.<br />
</pre><br />
<br />
NOTE: The default UHD FPGA Images destination within the E320's file-system is <code>/usr/share/uhd/images</code>. The default UHD FPGA Images destination on a typical host installation is <code>/usr/local/share/uhd/images</code>.<br />
<br />
Updating the FPGA image from the ARM CPU is the same as detailed above for a Network mode update:<br />
<br />
<pre><br />
root@ni-e320-serial:~# uhd_image_loader --args "type=e3xx,fpga=1G"<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106600; UHD_3.13.1.0-0-unknown<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=127.0.0.1,type=e3xx,product=e320,serial=316E375,claimed=False,skip_init=1<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPMD] Claimed device without full initialization.<br />
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.<br />
[INFO] [MPM.PeriphManager] Updating component `fpga'<br />
[INFO] [MPM.PeriphManager] Updating component `dts'<br />
[INFO] [MPM.RPCServer] Resetting peripheral manager.<br />
[INFO] [MPM.PeriphManager] Device serial number: 316E375<br />
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.<br />
[INFO] [MPM.PeriphManager] Found 1 daughterboard(s).<br />
<br />
</pre><br />
<br />
For more information on updating the FPGA image, refer to the UHD Manual at http://uhd.ettus.com.<br />
<br />
==Setting Up a Streaming Connection==<br />
The device supports multiple high-speed, low-latency interfaces on the SFP+ port for streaming samples to the host computer.<br />
<br />
===1 Gb Streaming via SFP+ Port ===<br />
Complete the steps below to set up a streaming connection over the 1 Gb Ethernet interface on the <code>SFP+ Port</code>.<br />
<br />
NOTE: The <code>1G</code> FPGA image must be loaded for the <code>SFP+ Port</code> to operate at 1 Gb speeds. If the <code>XG</code> image is loaded, the port will be unresponsive at 1Gb speeds.<br />
<br />
1. Configure your Host's 1 Gb Ethernet interface as shown below. This interface should be separate from the 1 Gb NIC/network which is connected to the 1 Gb RJ45 management interface. <br />
<br />
IP Address: 192.168.10.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 1500<br />
<br />
NOTE: When operating the <code>SFP+ Port</code> at 1 Gb speeds, it is important to set a MTU of <code>1500</code> and not a value of <code>automatic</code>. Mismatched MTU values on either the Host or E320 may cause flow control errors. Your computer may need to be restarted for the MTU value to take effect.<br />
<br />
2. Insert the RJ45-to-SFP+ adapter into the <code>SFP+ Port</code>.<br />
<br />
3. Connect the SFP+ adapter on the device to an Ethernet port on the host computer using a standard Ethernet cable.<br />
<br />
The Green LED above the <code>SFP+ Port</code> should illuminate.<br />
<br />
4. To test the connection, <code>ping</code> the device at address <code>192.168.10.2</code> from the host, as shown below:<br />
<br />
$ ping 192.168.10.2<br />
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.<br />
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=1.06 ms<br />
^C<br />
--- 192.168.10.2 ping statistics ---<br />
1 packets transmitted, 1 received, 0% packet loss, time 0ms<br />
rtt min/avg/max/mdev = 1.065/1.065/1.065/0.000 ms<br />
<br />
Press <code>CTRL+C</code> to stop the ping program.<br />
<br />
5. Verify your MTU is set correctly for 1 Gb speeds on the E320. See the section [[E320_Getting_Started_Guide#Updating_the_Network_Configurations|Updating the Network Configurations]] for additional details.<br />
<br />
Proceed to the next section [[E320_Getting_Started_Guide#Verifying_Device_Operation|Verifying Device Operation]].<br />
<br />
===10 Gb Streaming via SFP+ Port===<br />
Load the <code>XG</code> FPGA image for 10 Gb streaming as detailed in the section [[E320_Getting_Started_Guide#Updating_the_FPGA_Image|Updating the FPGA Image]]. You will need to use a 10 GigE cable that can be plugged in directly to the SFP+ connector on the board. <br />
<br />
NOTE: The <code>XG</code> FPGA image must be loaded for the <code>SFP+ Port</code> to operate at 10 Gb speeds. If the <code>1G</code> image is loaded, the port will be unresponsive at 10 Gb speeds. Mismatched MTU values on either the Host or E320 may cause flow control errors.<br />
<br />
1. Configure your Host's 10 Gb Ethernet interface as shown below. <br />
<br />
IP Address: 192.168.10.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 8000<br />
<br />
NOTE: When operating the <code>SFP+ Port</code> at 10 Gb speeds, it is important to set a MTU of <code>8000</code> and not a value of <code>automatic</code>. Mismatched MTU values on either the Host or E320 may cause flow control errors. Your computer may need to be restarted for the MTU value to take effect.<br />
<br />
2. Connect the SFP+ port on the device to an Ethernet port on the host computer using a 10 Gb SFP+ copper or fiber cable.<br />
<br />
The Green LED above the <code>SFP+ Port</code> should illuminate.<br />
<br />
4. To test the connection, <code>ping</code> the device at address <code>192.168.10.2</code> from the host, as shown below:<br />
<br />
$ ping 192.168.10.2<br />
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.<br />
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=1.06 ms<br />
^C<br />
--- 192.168.10.2 ping statistics ---<br />
1 packets transmitted, 1 received, 0% packet loss, time 0ms<br />
rtt min/avg/max/mdev = 1.065/1.065/1.065/0.000 ms<br />
<br />
Press <code>CTRL+C</code> to stop the ping program.<br />
<br />
5. Verify your MTU is set correctly for 10 Gb speeds on the E320. See the section [[E320_Getting_Started_Guide#Updating_the_Network_Configurations|Updating the Network Configurations]] for additional details.<br />
<br />
Proceed to the next section [[E320_Getting_Started_Guide#Verifying_Device_Operation|Verifying Device Operation]].<br />
<br />
==Verifying Device Operation==<br />
Once you have successfully setup a management interface and streaming interface, you can now verify the devices operation using the include UHD utilities.<br />
<br />
===Subdevice Specification Mapping===<br />
The USRP E320 contains 2 channels, each represented on the front panel as <code>RF A</code> and <code>RF B</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
'''E320'''<br />
* RF A = A:0<br />
* RF B = A:1<br />
<br />
Additional details of UHD Subdevice Specifications can be found here in the UHD Manual: http://files.ettus.com/manual/page_configuration.html#config_subdev<br />
<br />
===Supported Sample Rates===<br />
<br />
The USRP E320 supports master clock rate from 200 kHz to 61.44 MHz and can be changed by adding <code>master_clock_rate=<rate></code> to the default UHD args. The default master clock rate is 16 MHz. <br />
<br />
Sample rates as delivered to/from the host computer for USRP devices are constrained to follow several important rules.<br />
<br />
It is important to understand that strictly-integer decimation and interpolation are used within USRP hardware to meet the requested sample rate requirements of the application at hand. That means that the desired sample rate must meet the requirement that master-clock-rate/desired-sample-rate be an integer ratio. Further, it is strongly desirable for that ratio to be even. This ratio is the decimation (down-conversion) or interpolation (up-conversion) factor. The decimation or interpolation factor may be between 1 and 1024. There are further constraints on the decimation or interpolation factor. If the decimation or interpolation factor exceeds 128, then it must be evenly divisible by 2. If the decimation or interpolation factor exceeds 256, then it must be evenly divisible by 4.<br />
<br />
Additional information on Sample Rates can be found here in the UHD Manual: http://files.ettus.com/manual/page_general.html#general_sampleratenotes<br />
<br />
===Probe the USRP E320===<br />
The UHD utility <code>uhd_usrp_probe</code> provides detailed information of the USRP device.<br />
<br />
From your host computer, run the command <code>uhd_usrp_probe</code>:<br />
<br />
<pre><br />
$ uhd_usrp_probe --args "addr=192.168.10.2"<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.13.1.0-1-gd3b7e90a<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.10.2,type=e3xx,product=e320,serial=316E375,claimed=False,addr=192.168.10.2<br />
[INFO] [MPM.PeriphManager] init() called with device args `product=e320,mgmt_addr=192.168.10.2'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000000)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1343 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1335 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000003320)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/Radio_0] Performing CODEC loopback test... <br />
[INFO] [0/Radio_0] CODEC loopback test passed<br />
[INFO] [0/Radio_0] Performing CODEC loopback test... <br />
[INFO] [0/Radio_0] CODEC loopback test passed<br />
_____________________________________________________<br />
/<br />
| Device: E300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-e320-316E375<br />
| | eeprom_version: 2<br />
| | mpm_version: 3.13.1.0-gd3b7e90a<br />
| | pid: 58144<br />
| | product: e320<br />
| | rev: 2<br />
| | rpc_connection: remote<br />
| | serial: 316E375<br />
| | type: e3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 3.0<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_locked, temp_main_power, ref_locked, temp_rf_channelA, temp_fpga, gps_sky, temp_rf_channelB, fan, temp_internal, gps_tpv, gps_time<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Neon<br />
| | | | Antennas: RX2, TX/RX<br />
| | | | Sensors: lo_locked, ad9361_temperature, rssi, lo_lock<br />
| | | | Freq range: 70.000 to 6000.000 MHz<br />
| | | | Gain range PGA: 0.0 to 76.0 step 1.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 40000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 1<br />
| | | | Name: Neon<br />
| | | | Antennas: RX2, TX/RX<br />
| | | | Sensors: lo_locked, ad9361_temperature, rssi, lo_lock<br />
| | | | Freq range: 70.000 to 6000.000 MHz<br />
| | | | Gain range PGA: 0.0 to 76.0 step 1.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 40000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: AD9361 Dual ADC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Neon<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9361_temperature<br />
| | | | Freq range: 47.000 to 6000.000 MHz<br />
| | | | Gain range PGA: 0.0 to 89.8 step 0.2 dB<br />
| | | | Bandwidth range: 20000000.0 to 40000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 1<br />
| | | | Name: Neon<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9361_temperature<br />
| | | | Freq range: 47.000 to 6000.000 MHz<br />
| | | | Gain range PGA: 0.0 to 89.8 step 0.2 dB<br />
| | | | Bandwidth range: 20000000.0 to 40000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: AD9361 Dual DAC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * DmaFIFO_0<br />
| | | * Radio_0<br />
| | | * DDC_0<br />
| | | * DUC_0<br />
<br />
</pre><br />
<br />
<br />
<br />
===ASCII Art Example===<br />
The UHD driver includes several example programs, which may serve as test programs or the basis for your application program. The source code can be obtained from the UHD repository on github at: https://github.com/EttusResearch/uhd/tree/master/host/examples<br />
<br />
You can quickly verify the operation of your USRP E320 by running the <code>rx_ascii_art_dft</code> UHD example program.<br />
<br />
The <code>rx_ascii_art_dft</code> utility is a simple console based, real-time FFT display tool. It is not graphical in nature, so it can be easily run over an SSH connection within a terminal window, and does not need any graphical capability, such as X Windows, to be installed. It can also be run over a serial console connection, although this is not recommended, as the formatting may not render correctly.<br />
<br />
You can run a simple test of the E320 USRP by connecting an antenna and observing the spectrum of a commercial FM radio station in real-time, following the steps below:<br />
<br />
1. Attach an antenna to the <code>RF A / RX2</code> antenna port of the E320.<br />
<br />
2. From your host computer, run the command:<br />
<br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft \<br />
--args "addr=192.168.10.2" \<br />
--freq 98.5e6 \<br />
--rate 2e6 \<br />
--gain 40 \<br />
--ref-lvl="-30" \<br />
--dyn-rng 90 \<br />
--ant "RX2" \<br />
--subdev "A:0"<br />
<br />
NOTE: Modify the commandline argument <code>freq</code> above to specify a tuning frequency for a strong local FM radio station. You will also need to update the IP Address to match your device IP.<br />
<br />
3. You should see a real-time FFT display of 2 MHz of spectrum, centered at the specified tuning frequency.<br />
<br />
4. Type "<code>Q</code>" to stop the program and to return to the Linux command line.<br />
<br />
5. You can run with the <code>--help</code> argument to see a description of all available command-line options.<br />
<br />
Example Output:<br />
<br />
<pre><br />
$ ./rx_ascii_art_dft --args "addr=192.168.10.2" --freq 98.5e6 --rate 2e6 --gain 40 --ref-lvl="-30" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
<br />
Creating the usrp device with: addr=192.168.10.2...<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.13.1.0-1-gd3b7e90a<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.10.2,type=e3xx,product=e320,serial=316E375,claimed=False,addr=192.168.10.2<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000000)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1334 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1325 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000003320)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [MPM.PeriphManager] init() called with device args `product=e320,mgmt_addr=192.168.10.2'.<br />
[INFO] [0/Radio_0] Performing CODEC loopback test... <br />
[INFO] [0/Radio_0] CODEC loopback test passed<br />
[INFO] [0/Radio_0] Performing CODEC loopback test... <br />
[INFO] [0/Radio_0] CODEC loopback test passed<br />
Using Device: Single USRP:<br />
Device: E300-Series Device<br />
Mboard 0: ni-e320-316E375<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Neon<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Neon<br />
TX Channel: 1<br />
TX DSP: 1<br />
TX Dboard: A<br />
TX Subdev: Neon<br />
<br />
Setting RX Rate: 2.000000 Msps...<br />
Actual RX Rate: 2.000000 Msps...<br />
<br />
Setting RX Freq: 98.500000 MHz...<br />
Actual RX Freq: 98.500000 MHz...<br />
<br />
Setting RX Gain: 40.000000 dB...<br />
Actual RX Gain: 40.000000 dB...<br />
<br />
Checking RX: all_los: locked ...<br />
<br />
Done!<br />
</pre><br />
<br />
===Benchmarking your system===<br />
Included with the UHD driver example programs is a utility, <code>benchmark_rate</code> to benchmark the transport link of the system.<br />
<br />
A system's maximum performance is dependent upon many factors. <code>benchmark_rate</code> will exercise the transport link and CPU of the system.<br />
<br />
====1 Gb Interface====<br />
NOTE: This example requires the <code>1G</code> FPGA image to be loaded.<br />
<br />
This example will test one full-duplex stream using "RFA/A:0", at a rate of 2 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "addr=192.168.10.2" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 2e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 2e6 \<br />
--tx_subdev "A:0"<br />
<br />
This example will test two full-duplex streams at 2 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "addr=192.168.10.2" \<br />
--duration 60 \<br />
--channels "0,1" \<br />
--rx_rate 2e6 \<br />
--rx_subdev "A:0 A:1" \<br />
--tx_rate 2e6 \<br />
--tx_subdev "A:0 A:1"<br />
<br />
This example will test two full-duplex streams at 12.5 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "addr=192.168.10.2,master_clock_rate=25e6" \<br />
--duration 60 \<br />
--channels "0,1" \<br />
--rx_rate 12.5e6 \<br />
--rx_subdev "A:0 A:1" \<br />
--tx_rate 12.5e6 \<br />
--tx_subdev "A:0 A:1"<br />
<br />
When streaming samples over a 1 Gb transport link, the maximum accumulative rate for all channels is 25 MS/s with a <code>sc16</code> OTW format. To achieve higher streaming rates, it is recommended to use the 10 Gb interfaces.<br />
<br />
====10 Gb Interface ====<br />
NOTE: These examples require the <code>XG</code> FPGA image to be loaded.<br />
<br />
This example will test one full-duplex stream using "RFA/A:0", at a rate of 61.44 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "addr=192.168.10.2,master_clock_rate=61.44e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 61.44e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 61.44e6 \<br />
--tx_subdev "A:0" <br />
<br />
This example will test two full-duplex stream, at a rate of 30.72 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "addr=192.168.10.2,master_clock_rate=61.44e6" \<br />
--duration 60 \<br />
--channels "0,1" \<br />
--rx_rate 30.72e6 \<br />
--rx_subdev "A:0 A:1" \<br />
--tx_rate 30.72e6 \<br />
--tx_subdev "A:0 A:1"<br />
<br />
==USRP E320 Device Specific Operations==<br />
<br />
===Turning the Device Off/On===<br />
To avoid damaging the file system and causing any corruption, do not turn the device off with the power button without first shutting down the system. Use this command to cleanly and properly shut the system down:<br />
<br />
shutdown -h now<br />
<br />
=== Autoboot ===<br />
<br />
The USRP E320 can be configured to power on and boot automatically when power is applied. By default, autoboot is disabled on all USRPs that support it. To control autoboot on the USRP E320, first determine the current value for <code>MCU_FLAGS[0]</code> by running <code>eeprom-dump</code>; the meaning of the <code>MCU_FLAGS[0]</code> [https://files.ettus.com/manual/page_usrp_e3xx.html#e320_eeprom_flags is found in the UHD manual]. The least significant bit when <code>MCU_FLAGS[0]</code> is viewed as a binary value controls the autoboot.<br />
<br />
For example<br />
<pre><br />
root@ni-e320-XXXXXXX:~# eeprom-dump<br />
-- PID/REV: e320 0002<br />
-- MCU_FLAGS[0]: 00000008<br />
-- MCU_FLAGS[1]: 00000000<br />
-- MCU_FLAGS[2]: 00000000<br />
-- MCU_FLAGS[3]: 00000000<br />
-- Serial: XXXXXXX<br />
-- eth_addr0: XX:XX:XX:XX:XX:XX<br />
-- eth_addr1: XX:XX:XX:XX:XX:XX<br />
-- eth_addr2: XX:XX:XX:XX:XX:XX<br />
-- DT-Compat/MCU-Compat: 0000 0002<br />
-- CRC: cbd79a61 (matches)<br />
</pre><br />
shows <code>-- MCU_FLAGS[0]: 00000008</code>; <code>0x08</code> (<code>0b00001000</code> in binary) indicates that autoboot is disabled. If this value were <code>0x09</code> (<code>0b00001001</code> in binary) it would indicate that autoboot is enabled because least significant bit is 1; same would be true if this value is <code>0x01</code> (<code>0b00000001</code> in binary).<br />
<br />
To enable or disable autoboot, copy the existing value of <code>MCU_FLAGS[0]</code> retrieved by <code>eeprom-dump</code> into <code><MCU_FLAGS[0]></code> below and run the command:<br />
<br />
* Disable autoboot on USRP E320 (sets least significant bit to 0), regardless of whether currently enabled or disabled:<br />
<pre><br />
root@ni-e320-XXXXXXX:~# eeprom-set-flags $((0x<MCU_FLAGS[0]> & ~0x1))<br />
</pre><br />
Thus, for the value noted above (autoboot is already disabled, so this command doesn't actually change anything):<br />
<pre><br />
root@ni-e320-XXXXXXX:~# eeprom-set-flags $((0x00000008 & ~0x1))<br />
-- PID/REV: e320 0002<br />
-- MCU_FLAGS[0]: 00000008<br />
-- MCU_FLAGS[1]: 00000000<br />
-- MCU_FLAGS[2]: 00000000<br />
-- MCU_FLAGS[3]: 00000000<br />
-- Serial: XXXXXXX<br />
-- eth_addr0: XX:XX:XX:XX:XX:XX<br />
-- eth_addr1: XX:XX:XX:XX:XX:XX<br />
-- eth_addr2: XX:XX:XX:XX:XX:XX<br />
-- DT-Compat/MCU-Compat: 0000 0002<br />
-- CRC: cbd79a61 (matches)<br />
-- Reading back <br />
-- PID/REV: e320 0002<br />
-- MCU_FLAGS[0]: 00000008<br />
-- MCU_FLAGS[1]: 00000000<br />
-- MCU_FLAGS[2]: 00000000<br />
-- MCU_FLAGS[3]: 00000000<br />
-- Serial: XXXXXXX<br />
-- eth_addr0: XX:XX:XX:XX:XX:XX<br />
-- eth_addr1: XX:XX:XX:XX:XX:XX<br />
-- eth_addr2: XX:XX:XX:XX:XX:XX<br />
-- DT-Compat/MCU-Compat: 0000 0002<br />
-- CRC: 448fb572 (matches)<br />
</pre><br />
<br />
* Enable autoboot on USRP E320 (sets least significant bit to 1), regardless of whether currently enabled or disabled. For example when changing from autoboot disabled to enabled:<br />
<pre><br />
root@ni-e320-XXXXXXX:~# eeprom-set-flags $((0x<MCU_FLAGS[0]> | 0x1))<br />
</pre><br />
Thus, for the value noted above:<br />
<pre><br />
root@ni-e320-XXXXXXX:~# eeprom-set-flags $((0x00000008 | 0x1))<br />
-- PID/REV: e320 0002<br />
-- MCU_FLAGS[0]: 00000008<br />
-- MCU_FLAGS[1]: 00000000<br />
-- MCU_FLAGS[2]: 00000000<br />
-- MCU_FLAGS[3]: 00000000<br />
-- Serial: XXXXXXX<br />
-- eth_addr0: XX:XX:XX:XX:XX:XX<br />
-- eth_addr1: XX:XX:XX:XX:XX:XX<br />
-- eth_addr2: XX:XX:XX:XX:XX:XX<br />
-- DT-Compat/MCU-Compat: 0000 0002<br />
-- CRC: cbd79a61 (matches)<br />
-- Reading back <br />
-- PID/REV: e320 0002<br />
-- MCU_FLAGS[0]: 00000009<br />
-- MCU_FLAGS[1]: 00000000<br />
-- MCU_FLAGS[2]: 00000000<br />
-- MCU_FLAGS[3]: 00000000<br />
-- Serial: XXXXXXX<br />
-- eth_addr0: XX:XX:XX:XX:XX:XX<br />
-- eth_addr1: XX:XX:XX:XX:XX:XX<br />
-- eth_addr2: XX:XX:XX:XX:XX:XX<br />
-- DT-Compat/MCU-Compat: 0000 0002<br />
-- CRC: 448fb572 (matches)<br />
</pre><br />
<br />
If setting this flag ''does not'' allow autoboot control on the USRP E320, then the device boot firmware needs to be updated. This update is accomplished via the following instructions.<br />
<br />
On the USRP E320 via ssh or serial terminal, [https://files.ettus.com/binaries/misc/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7.tar.gz download the update MCU firmware] and extract it:<br />
<pre><br />
root@ni-e320-XXXXXXX:~# curl https://files.ettus.com/binaries/misc/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7.tar.gz | tar zxf -<br />
</pre><br />
This will create a directory <code>upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7</code>. Go into this directory and run the firmware flash script:<br />
<pre><br />
root@ni-e320-XXXXXXX:~# cd upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7<br />
root@ni-e320-XXXXXXX:~/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7# ./flash-firmware.sh<br />
This script updates the microcontroller firmware (RO part). The change is<br />
persistent across power cycles. Incorrect updates can only fixed be a manual<br />
process which requires opening the enclosure.<br />
<br />
Updating the microcontroller firmware (RO part) is only required if the Ettus<br />
Research support told you to do so.<br />
<br />
Press "y" to continue<br />
</pre><br />
At the prompt, press the <code>y</code> key to continue. Pressing any other key aborts the procedure:<br />
<pre><br />
Press "y" to continue n<br />
<br />
aborting<br />
root@ni-e320-317F9BF:~/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7# <br />
</pre><br />
Pressing the <code>y</code> key:<br />
<pre><br />
Press "y" to continue y<br />
<br />
This script will flash ec-neon-rev3.RO.flat to the device<br />
old RO version: neon_vX.X.XXXX-XXXXXXX<br />
new RO version: neon_v1.1.7358-a190641<br />
<br />
Press "y" to continue<br />
</pre><br />
At the prompt, press the <code>y</code> key ''again'' to continue. Pressing any other key aborts the procedure as before.<br />
<pre><br />
Press "y" to continue y<br />
<br />
./ectool --interface=dev reboot_ec RW<br />
./ectool --interface=dev flashread 0x0 65536 ec-neon-rev3.RO.flat.old<br />
Reading 65536 bytes at offset 0...<br />
done.<br />
./ectool --interface=dev flasherase 0x0 65536<br />
Erasing 65536 bytes at offset 0...<br />
done.<br />
./ectool --interface=dev flashwrite 0x0 ec-neon-rev3.RO.flat<br />
Reading 49592 bytes from ec-neon-rev3.RO.flat...<br />
Writing to offset 0...<br />
Write size 112...<br />
done.<br />
<br />
copying new firmware files<br />
'ec-neon-rev3.bin' -> '/lib/firmware/ni/ec-neon-rev3.bin'<br />
'ec-neon-rev3.RW.bin' -> '/lib/firmware/ni/ec-neon-rev3.RW.bin'<br />
root@ni-e320-317F9BF:~/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7# <br />
</pre><br />
<br />
Once the script is done, reboot the USRP (e.g., <code>shutdown -r now</code>), and when it comes up the autoboot flag should now work as desired. If these instructions ''do not'' work, then email [mailto:support@ettus.com support@ettus.com] and ask for alternative instructions on how to update the USRP E320 RO and RW boot firmware such that this EEPROM flag setting is honored.<br />
<br />
===Default Password===<br />
The default user is <code>root</code> and the password is empty (no password).<br />
<br />
It is recommended to update the <code>root</code> password, which can be done with the command <code>passwd</code>:<br />
<br />
Example Output:<br />
<br />
root@ni-e320-serial:~# passwd<br />
Changing password for root<br />
New password:<br />
Re-enter new password:<br />
passwd: password changed.<br />
<br />
==Known Issues==<br />
===Problematic NICs===<br />
In some streaming modes, the Intel I219-LM NIC can produce flow control and sequence errors. It is recommended to use a USB3 to 1 Gb Ethernet Adapter for hosts which have an I219-LM NIC.<br />
<br />
==Technical Support and Community Knowledge Base==<br />
Technical support for USRP hardware is available through email only. If the product arrived in a nonfunctional state or you require technical assistance, please contact [mailto:support@ettus.com support@ettus.com]. Please allow 24 to 48 hours for response by email, depending on holidays and weekends, although we are often able to reply more quickly than that.<br />
<br />
We also recommend that you subscribe to the community mailing lists. The mailing lists have a responsive and knowledgeable community of hundreds of developers and technical users who are located around the world. When you join the community, you will be connected to this group of people who can help you learn about SDR and respond to your technical and specific questions. Often your question can be answered quickly on the mailing lists. Each mailing list also provides an archive of all past conversations and discussions going back many years. Your question or problem may have already been addressed before, and a relevant or helpful solution may already exist in the archive.<br />
<br />
Discussions involving the USRP hardware and the UHD software itself are best addressed through the '''usrp-users''' mailing list at [http://usrp-users.ettus.com http://usrp-users.ettus.com].<br />
<br />
Discussions involving the use of [http://gnuradio.org/ GNU Radio] with USRP hardware and UHD software are best addressed through the '''discuss-gnuradio''' mailing list at [https://lists.gnu.org/mailman/listinfo/discussgnuradio https://lists.gnu.org/mailman/listinfo/discussgnuradio].<br />
<br />
Discussions involving the use of [http://openbts.org/ OpenBTS®] with USRP hardware and UHD software are best addressed through the '''openbts-discuss''' mailing list at [https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss].<br />
<br />
The support page on our website is located at [https://www.ettus.com/support https://www.ettus.com/support]. The Knowledge Base is located at [https://kb.ettus.com https://kb.ettus.com].<br />
<br />
==Legal Considerations==<br />
Every country has laws governing the transmission and reception of radio signals. Users are solely responsible for insuring they use their USRP system in compliance with all applicable laws and regulations. Before attempting to transmit and/or receive on any frequency, we recommend that you determine what licenses may be required and what restrictions may apply.<br />
<br />
*NOTE: This USRP product is a piece of test equipment.<br />
<br />
==Sales and Ordering Support==<br />
If you have any non-technical questions related to your order, then please contact us by email at [mailto:orders@ettus.com orders@ettus.com], or by phone at +14086106399 (Monday-Friday, 8 AM - 5 PM, Pacific Time). Please be sure to include your order number and the serial number of your USRP.<br />
<br />
==Terms and Conditions of Sale==<br />
Terms and conditions of sale can be accessed online at the following link: http://www.ettus.com/legal/terms-and-conditions-of-sale<br />
<br />
[[Category:Getting Started Guides]]<br />
[[Category:E320]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=USRP_N300/N310/N320/N321_Getting_Started_Guide&diff=6025USRP N300/N310/N320/N321 Getting Started Guide2024-03-11T19:50:36Z<p>MichaelDickens: /* Mender Update Process */ add mender output progress info</p>
<hr />
<div>==Kit Contents==<br />
===N300===<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N300<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n300 kit.png|450px|center]]<br />
|}<br />
===N310===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N310<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n310 kit.png|500px|center]]<br />
|}<br />
<br />
===N320===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N320<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
|[[File:n320 kit.png|500px|center]] <br />
|}<br />
<br />
===N321===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N321<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n321 kit.png|500px|center]] <br />
|}<br />
==Verify the Contents of Your Kit==<br />
Ensure that your kit contains all the items listed above. If any items are missing, please contact sales@ettus.com immediately.<br />
<br />
==You Will Need==<br />
* microSD Card Writer<br />
<br />
* For Network Mode: A host computer with an available 1 or 10 Gigabit Ethernet interface for sample streaming. In addition to the Ethernet interface used for sampling streaming, your host computer will require a separate 1 Gigabit Ethernet interface for command and control streaming.<br />
<br />
* For Stand-Alone Embedded Mode: A host computer with an available 1 Gigabit Ethernet port or a USB 2.0 port to remotely access the embedded Linux operating system running on ARM CPU.<br />
<br />
==Proper Care and Handling==<br />
All Ettus Research products are individually tested before shipment. The USRP is guaranteed to be functional at the time it is received by the customer. Improper use or handling of the USRP can cause the device to become non-functional. Take the following precautions to prevent damage to the unit.<br />
<br />
* Never allow metal objects to touch the circuit board while powered.<br />
* Always properly terminate the transmit port with an antenna or 50Ω load.<br />
* Always handle the board with proper anti-static methods.<br />
* Never allow the board to directly or indirectly come into contact with any voltage spikes.<br />
* Never allow any water or condensing moisture to come into contact with the device.<br />
* Always use caution with FPGA, firmware, or software modifications.<br />
<br />
{|<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |Never apply more than -15 dBm of power into any RF input.<br />
|-<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |Always use at least 30dB attenuation if operating in loopback configuration<br />
|-<br />
|}<br />
<br />
==Install and Setup the Software Tools on Your Host Computer==<br />
In order to use your Universal Software Radio Peripheral (USRP™), you must have the software tools correctly installed and configured on your host computer. A step-by-step guide for doing this is available at the Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux|Linux]], [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X|OS X]] and [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows|Windows]] Application Notes.<br />
<br />
To find the latest release of UHD, see the UHD repository at https://github.com/EttusResearch/uhd.<br />
<br />
The USRP N310 requires UHD version 3.11.0.0 or later. <br />
<br />
The USRP N300 requires UHD version 3.12.0.0 or later.<br />
<br />
The USRP N320/N321 requires UHD version 3.14.0.0 or later. <br />
<br />
White Rabbit Ethernet-Based Synchronization of the N3xx USRP requires UHD version 3.12.0.0 or later. For additional details on White Rabbit Ethernet-Based Synchronization, please see the application note, [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]].<br />
<br />
'''When you receive a brand-new device, it is strongly recommended that you download the most recent filesystem image from the Ettus Research website and write it to the SD card that comes with the unit. It is not recommended that you use the SD card from the factory as-is. Instructions on downloading the latest filesystem image and writing it to the SD card are listed below.'''<br />
<br />
'''Note that if you are operating the device in Network Mode, the version of UHD running on the host computer and the USRP N3xx must match.'''<br />
<br />
==Connecting the Device==<br />
===Interfaces Overview===<br />
Listed below are the interfaces to connect to the USRP N3xx. Each interface has specific functionality, limitations and purpose. <br />
<br />
'''Serial Console'''<br />
<br />
The Serial Console provides a low level interface to the device typically used for debugging.<br />
<br />
'''1 Gigabit RJ45 Connection'''<br />
<br />
The 1 Gigabit RJ45 Connection interfaces with the on-board ARM CPU. When operated in "Network mode", this interface can optionally be used for UHD management traffic. Regardless of the operation mode (Network vs Embedded) this interface can be used to connect to the ARM via SSH. By default, the 1Gb RJ45 connection is configured to use a DHCP assigned IP address.<br />
<br />
'''Dual SFP+ Connections'''<br />
<br />
The Dual SFP+ Connections support multiple configurations for streaming high-speed, low-latency data, depending upon the FPGA image which is loaded.<br />
<br />
'''QSFP+ Connection (N320/ N321 Only)'''<br />
<br />
The QSFP+ Connection supports 2 x 10Gb lanes for streaming high-speed, low-latency data, while the onboard SFP0 connection is used for White Rabbit Ethernet-Based Synchronization.<br />
<br />
===Setting up a Serial Console Connection===<br />
It is possible to gain shell access to the device using a serial terminal emulator via the Serial Console port. Most Linux, OSX, or other Unix based operating systems have a tool called <code>screen</code> which can be used for this purpose. <br />
<br />
If you do not have <code>screen</code> installed, it can be installed via your package manager. For Ubuntu/Debian based operating systems it can be installed with <code>apt</code> such as:<br />
<br />
sudo apt install screen<br />
<br />
The default Baud Rate for the Serial Console is: <code>115200</code><br />
<br />
The exact device node you should attach to depends on your operating system's driver and other USB devices that might already be connected. Modern Linux systems offer alternatives to simply trying device nodes; instead, the OS might have a directory of symlinks under <code>/dev/serial/by-id</code>:<br />
<br />
$ ls /dev/serial/by-id<br />
usb-Digilent_Digilent_USB_Device_25163511FE00-if00-port0<br />
usb-Digilent_Digilent_USB_Device_25163511FE00-if01-port0<br />
usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if00-port0<br />
usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if01-port0<br />
<br />
NOTE: Exact names depend on the host operating system version and may differ.<br />
<br />
Every N3XX series device connected to USB will by default show up as four different devices. The devices labeled <code>"USB_to_UART_Bridge_Controller"</code> are the devices that offer a serial prompt. The first (with the <code>if00</code> suffix) connects to the <code>ARM CPU</code>, whereas the second connects to the <code>STM32 Microcontroller</code>. <br />
<br />
If you have multiple N3xx Serial Consoles connected to a single host, you may have to empirically test nodes. <br />
<br />
Connecting to the ARM CPU can be performed with the command:<br />
<br />
$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if00-port0 115200<br />
<br />
Upon starting the USRP N3xx, boot messages will appear and rapidly update. Once the boot process successfully completes, a login prompt like the following should appear:<br />
<br />
OpenEmbedded test ni-n3xx-313ABDA ttyPS0<br />
<br />
ni-n3xx-313ABDA login: <br />
<br />
Enter the username: <code>root</code> <br />
<br />
By default, the <code>root</code> user's password is left blank. Press the <code>Enter</code> key when prompted for a password.<br />
<br />
You should now be presented with a shell prompt similar to the following:<br />
<br />
root@ni-n3xx-<motherboard serial #>:~#<br />
<br />
Using the default configuration, the serial console will show all kernel log messages (which are not available when using SSH), and give access to the boot loader (U-boot prompt). This can be used to debug kernel or boot-loader issues more efficiently than when logged in via SSH.<br />
<br />
====Connecting to the microcontroller====<br />
<br />
Using the Serial Console interface, it is possible to connect to the STM32 microcontroller with the command below. The STM32 controls the power sequencing and several other low level device operations.<br />
<br />
$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if01-port0 115200<br />
<br />
The STM32 interface provides a very simple prompt. The command <code>help</code> will list all available commands. A direct connection to the microcontroller can be used to hard-reset the device without physically accessing it (i.e., emulating a power button press) and other low-level diagnostics.<br />
<br />
===Connecting to the ARM via SSH===<br />
By default, the RJ45 1Gb management interface is configured to be assigned a DHCP IP address. <br />
<br />
If you have access to a network which provides a DHCP server (such as a common router's LAN), attach the RJ45 1Gb port to this network. Details vary by vendor, however, most router management interfaces will provide a list of attached devices to the LAN including their IP address.<br />
<br />
Without access to a router management interface, you can identify the IP address by connecting to the ARM CPU via Serial Console as detailed in the section above and running the command <code>ip a</code>:<br />
<br />
Example Output:<br />
<br />
<pre><br />
# ip a<br />
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000<br />
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br />
inet 127.0.0.1/8 scope host lo<br />
valid_lft forever preferred_lft forever<br />
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
inet 192.168.1.151/24 brd 192.168.1.255 scope global dynamic eth0<br />
valid_lft 42865sec preferred_lft 42865sec<br />
3: sfp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
inet 192.168.10.2/24 brd 192.168.10.255 scope global sfp0<br />
valid_lft forever preferred_lft forever<br />
4: sfp1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
</pre><br />
<br />
If you do not have access to a network with a DHCP server, you can create one using the Linux utility <code>dnsmasq</code>:<br />
<br />
$ sudo dnsmasq -i <ETHERNET_ADAPTER_NAME> --dhcp-range=192.168.1.151,192.168.1.254 --except-interface=lo --bind-dynamic --no-daemon<br />
<br />
NOTE: Modify the value <code><ETHERNET_ADAPTER_NAME></code> to match the interface you would like to create a DHCP server on.<br />
<br />
After the device has obtained an IP address, you can remotely log into it from a Linux or macOS system with SSH, as shown below:<br />
<br />
$ ssh root@192.168.1.151<br />
<br />
NOTE: The IP address may vary depending on your network setup.<br />
<br />
NOTE: The <code>root</code> password default password is empty/blank.<br />
<br />
On Microsoft Windows, the SSH connection can be established using the third-party program Putty. <br />
<br />
After logging in, you should be presented with a shell like the following:<br />
<br />
root@ni-n3xx-<motherboard serial #>:~#<br />
<br />
==Updating the Linux File System==<br />
Before operating the device, it is strongly recommended to update to the latest version of the Embedded Linux file system. If you are operating the device in Network Mode, the version of UHD running on the host machine and N3xx USRP must match. <br />
<br />
There is two ways to update the file system for the N3xx USRP: <br />
<br />
1. Mender<br />
<br />
2. Physically remove microSD card from device and write a new file system to the microSD card. <br />
<br />
===File System Partition Layout===<br />
The SD Card is divided into four partitions. There is two root file system partitions, a boot partition and a data partition. <br />
<br />
Any data you would like to preserve through Mender updates should be saved to the <code>data</code> partition, which is mounted at <code>/data</code>.<br />
<br />
===Updating the file system with Mender===<br />
Mender is third-party software that enables remote updating of the root file system without physically accessing the device (see also the Mender website https://mender.io). Mender can be executed locally on the device, or a Mender server can be set up which can be used to remotely update an arbitrary number of USRP devices. Users can host their own local Mender server, or use servers hosted by Mender as a paid service; contact Mender for more information. <br />
<br />
====Mender Update Process====<br />
<br />
When updating the file system using Mender, the tool will overwrite the root file system partition that is not currently mounted. Any data stored in the root partitions will be permanently lost with a Mender update.<br />
<br />
After updating a partition with Mender, it will reboot into the newly updated partition. Only if the update is confirmed by the user, the update will be made permanent. This means that if an update fails, the device will be always able to reboot into the partition from which the update was originally launched, which presumably is in a working state. Another update can be launched now to correct the previous, failed update, until it works.<br />
<br />
To obtain the file system Mender image (these are files with a <code>.mender</code> suffix), run the following command on the host computer with Internet access:<br />
$ sudo uhd_images_downloader -t mender -t n3xx --yes<br />
<br />
Example Output:<br />
[INFO] Using base URL: https://files.ettus.com/binaries/cache/<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
365684 kB / 365684 kB (100%) n3xx_common_mender_default-v4.6.0.0.zip<br />
[INFO] Images download complete.<br />
<br />
The downloaded "zip" archive is extracted into the <code>Images destination</code> directory with the filename <code>usrp_n3xx_fs.mender</code>. Next, you will need to copy this Mender file system image from the <code>Images destination</code> directory to the USRP N3xx to have its filesystem changed. This can be done with the Linux utility <code>scp</code>, for example as follows:<br />
<br />
$ scp /usr/local/share/uhd/images/usrp_n3xx_fs.mender root@192.168.1.51:~/. <br />
<br />
Note: The path and IP may be different for your configuration; the command above assumes you're using the default UHD installation path of <code>/usr/local</code> and that the N3xx's IP is <code>192.168.1.51</code>.<br />
<br />
After copying the Mender file system image to the N3xx, connect to the N3xx to gain shell access via either the Serial Console or SSH.<br />
<br />
On the N3xx, you first need to determine the version of UHD currently running on the USRP; an easy way to do this is via the command<br />
# uhd_config_info --version<br />
<br />
Example output:<br />
UHD 3.14.1.1-0-g0347a6d8<br />
<br />
The mender command to execute is different for UHD version 4.0 or newer versus prior to version 4.0. For the former use <code>mender install</code> followed by the mender file; for the latter use <code>mender -f -rootfs</code> followed by the mender file. Starting with UHD version 4.0 one can use mender to upgrade or downgrade the UHD filesystem version between any UHD v4 versions (e.g., 4.1 to 4.6; 4.6 to 4.1). The following commands assume that the UHD filesystem is version 4; if not then substitute the other mender command.<br />
<br />
Run <code>mender install /path/to/latest.mender</code> to update the file system, e.g.:<br />
<br />
# mender install usrp_n3xx_fs.mender<br />
<br />
The artifact can also be stored on a remote server:<br />
# mender install <nowiki>http://server.name/path/to/latest.mender</nowiki><br />
<br />
This procedure will take a quite a few minutes to complete. While executing, mender will show progress, e.g.:<br />
................................ 0% 1024 KiB<br />
...<br />
................................ 1% 4096 KiB<br />
...<br />
................................ 99% 386048 KiB<br />
............ 100% 386458 KiB<br />
INFO[3865] wrote 7851737088/7851737088 bytes of update to device /dev/mmcblk0p3 module=device<br />
INFO[3871] Enabling partition with new image installed to be a boot candidate: 3 module=device<br />
<br />
After mender has logged a successful update, reboot the device:<br />
# reboot<br />
<br />
Upon reboot log back in to the USRP N3xx and run the command <code>uhd_find_devices</code> to verify that the UHD version is as desired and that the command runs successfully -- it should find at a minimum the USRP it is being executed on and will find more if other USRPs are on the same network.<br />
<br />
If upon reboot the device is not working or the UHD version is not as desired, then the easiest way forward is to [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide#Updating_the_files_system_by_writing_the_disk_image overwrite the sdcard filesystem manually] with the desired UHD version.<br />
<br />
If upon reboot everything is as desired and the device seems functional, then commit the changes so that the boot loader knows to permanently boot into this partition:<br />
# mender commit<br />
<br />
To identify the currently installed Mender artifact from the command line, the following file can be queried on the N3xx:<br />
# cat /etc/mender/artifact_info<br />
<br />
If you are using a Mender server, the updates can be initiated from a web dashboard. From there, you can start the updates without having to log into the device, and you can update groups of USRPs with a few clicks in a web GUI. The dashboard can also be used to inspect the state of USRPs. This is a simple way to update groups of rack-mounted USRPs with custom file systems.<br />
<br />
For more information on updating the filesystem, refer to the [https://files.ettus.com/manual/ UHD Manual].<br />
<br />
===Updating the files system by writing the disk image===<br />
Please see the separate application note, [[Writing the USRP File System Disk Image to a SD Card]], for step-by-step instructions on writing the file system image to the SD card.<br />
<br />
==Updating the Network Configurations==<br />
The USRP N3xx systemd network configuration files are located at: <code>/data/network/</code><br />
<br />
# ls /data/network/<br />
eth0.network int0.network sfp0.network sfp1.network<br />
<br />
or for older versions of the file system: <code>/etc/systemd/network/</code><br />
<br />
# ls /etc/systemd/network/<br />
eth0.network sfp0.network sfp1.network<br />
<br />
For details on configuration please refer to the [https://www.freedesktop.org/software/systemd/man/systemd.network.html systemd-networkd manual pages].<br />
<br />
The factory settings are as follows:<br />
<pre><br />
eth0 (DHCP):<br />
<br />
[Match]<br />
Name=eth0<br />
<br />
[Network]<br />
DHCP=ipv4<br />
<br />
[DHCP]<br />
UseHostname=false<br />
<br />
sfp0 (static):<br />
<br />
[Match]<br />
Name=sfp0<br />
<br />
[Network]<br />
Address=192.168.10.2/24<br />
<br />
[Link]<br />
MTUBytes=9000<br />
<br />
sfp1 (static):<br />
<br />
[Match]<br />
Name=sfp1<br />
<br />
[Network]<br />
Address=192.168.20.2/24<br />
<br />
[Link]<br />
MTUBytes=9000<br />
</pre><br />
<br />
Additional notes on networking:<br />
<br />
* Care needs to be taken when editing these files on the device, since <code>vi</code> / <code>vim</code> sometimes generates undo files (e.g. <code>/etc/systemd/network/sfp0.network~</code>), that <code>systemd-networkd</code> might accidentally pick up.<br />
* Temporarily setting the IP addresses or MTU sizes via <code>ifconfig</code> or other command line tools will only change the value until the next reboot or reload of the FPGA image.<br />
* If the MTU of the device and host computers differ, streaming issues can occur.<br />
* Streaming via SFP0 at 1 Gb rates requires a MTU of <code>1500</code><br />
* Streaming via SFP0 at 10 Gb rates requires a MTU of <code>9000</code><br />
<br />
For addition details on network configuration here: https://files.ettus.com/manual/page_usrp_n3xx.html#n3xx_network_configuration<br />
<br />
==Updating the FPGA Image==<br />
<br />
===Network Mode FPGA Image Update===<br />
The FPGA image should match the version of UHD installed on the host computer, when operated in Network mode. Connect the device to the host computer using either the RJ45 or SFP+ port, refer to the section above for detailed instructions. <br />
<br />
To obtain all the FPGA images for a specific version of UHD, run the following command on the host computer with internet access:<br />
<br />
$ sudo uhd_images_downloader<br />
<br />
Example Output:<br />
<br />
$ sudo uhd_images_downloader<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
00006 kB / 00006 kB (100%) usrp1_b100_fw_default-g6bea23d.zip<br />
19810 kB / 19810 kB (100%) x3xx_x310_fpga_default-gf1ba32fe.zip<br />
02757 kB / 02757 kB (100%) usrp2_n210_fpga_default-g6bea23d.zip<br />
02123 kB / 02123 kB (100%) n230_n230_fpga_default-ge57dfe0.zip<br />
00522 kB / 00522 kB (100%) usrp1_b100_fpga_default-g6bea23d.zip<br />
00491 kB / 00491 kB (100%) b2xx_b200_fpga_default-ge57dfe0.zip<br />
02415 kB / 02415 kB (100%) usrp2_n200_fpga_default-g6bea23d.zip<br />
08988 kB / 08988 kB (100%) e3xx_e320_fpga_default-g3de8954a.zip<br />
23045 kB / 23045 kB (100%) n3xx_n310_fpga_default-g3de8954a.zip<br />
00523 kB / 00523 kB (100%) b2xx_b205mini_fpga_default-ge57dfe0.zip<br />
18937 kB / 18937 kB (100%) x3xx_x300_fpga_default-gf1ba32fe.zip<br />
00017 kB / 00017 kB (100%) octoclock_octoclock_fw_default-g14000041.zip<br />
00007 kB / 00007 kB (100%) usrp2_usrp2_fw_default-g6bea23d.zip<br />
00009 kB / 00009 kB (100%) usrp2_n200_fw_default-g6bea23d.zip<br />
00450 kB / 00450 kB (100%) usrp2_usrp2_fpga_default-g6bea23d.zip<br />
00144 kB / 00144 kB (100%) b2xx_common_fw_default-ga69ab0c.zip<br />
25107 kB / 25107 kB (100%) n3xx_n320_fpga_default-g3de8954a.zip<br />
00464 kB / 00464 kB (100%) b2xx_b200mini_fpga_default-ge57dfe0.zip<br />
00319 kB / 00319 kB (100%) usrp1_usrp1_fpga_default-g6bea23d.zip<br />
04839 kB / 04839 kB (100%) usb_common_windrv_default-g14000041.zip<br />
00009 kB / 00009 kB (100%) usrp2_n210_fw_default-g6bea23d.zip<br />
16065 kB / 16065 kB (100%) n3xx_n300_fpga_default-g3de8954a.zip<br />
05578 kB / 05578 kB (100%) e3xx_e310_fpga_default-g4bc2c6f.zip<br />
00885 kB / 00885 kB (100%) b2xx_b210_fpga_default-ge57dfe0.zip<br />
[INFO] Images download complete.<br />
<br />
<br />
NOTE: In the above example output, the Images Destination folder is printed:<br />
<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
<br />
To list the N3xx FPGA images with a full path, run the command:<br />
<br />
$ ls -w 1 /usr/local/share/uhd/images/usrp_n3*.bit<br />
<br />
/usr/local/share/uhd/images/usrp_n300_fpga_AA.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_AA.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_AQ.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_XQ.bit<br />
<br />
To update the default <code>HG</code> variant of FPGA image, run the command:<br />
<br />
$ uhd_image_loader --args "type=n3xx,addr=<N3xx_IP_ADDR>,fpga=HG"<br />
<br />
Example Output:<br />
<br />
uhd_image_loader --args "type=n3xx,addr=192.168.1.151,fpga=HG"<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.11.1.HEAD-0-gad6b0935<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.151,type=n3xx,product=n310,serial=313ABDA,claimed=False,skip_init=1<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.11.1.0-gunknown<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPMD] Claimed device without full initialization.<br />
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.<br />
[INFO] [MPM.PeriphManager] Updating component `fpga'<br />
[INFO] [MPM.PeriphManager] Updating component `dts'<br />
[INFO] [MPM.RPCServer] Resetting peripheral manager.<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.<br />
<br />
<br />
To load a different default FPGA image (i.e. <code>XG</code>, <code>WG</code>), modify the device argument <code>fpga=</code> to a value of <code>fpga=XG</code> or <code>fpga=WG</code>.<br />
<br />
To specify the path to a custom FPGA image, use the <code>--fpga-path</code> argument. <br />
<br />
$ uhd_image_loader --args "type=n3xx,addr=<N3xx_IP_ADDR>" --fpga-path=/path/to/custom/fpga.bit<br />
<br />
The Verilog code for the FPGA in the USRP N3xx is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
<br />
===Embedded Mode FPGA Image Update===<br />
<br />
It is possible to update the FPGA image when operated in Embedded mode. Connect to the ARM CPU via Serial Console or SSH as detailed in the section above. <br />
<br />
Updating the FPGA image from the ARM CPU is the same as detailed above for a Network mode update, except it is not required to provide an <code>addr</code> device argument. <br />
<br />
uhd_image_loader --args "type=n3xx,fpga=HG"<br />
<br />
<pre><br />
root@ni-n3xx-313ABDA:~# uhd_image_loader --args "type=n3xx,fpga=HG"<br />
[INFO] [UHD] linux; GNU C++ version 7.2.0; Boost_106400; UHD_3.11.1.0-0-unknown<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=127.0.0.1,type=n3xx,product=n310,serial=313ABDA,claimed=False,skip_init=1<br />
[INFO] [MPMD] Claimed device without full initialization.<br />
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.<br />
[INFO] [MPM.PeriphManager] Updating component `fpga'<br />
[INFO] [MPM.PeriphManager] Updating component `dts'<br />
[INFO] [MPM.RPCServer] Resetting peripheral manager.<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.<br />
</pre><br />
<br />
For more information on updating the FPGA image, refer to the UHD Manual at http://uhd.ettus.com .<br />
<br />
==Setting Up a Streaming Connection==<br />
The device supports multiple, high-speed, low-latency interfaces on the SFP+ ports for streaming samples to the host computer. <br />
<br />
===1Gb Streaming SFP Port 0===<br />
Complete the steps below to set up a streaming connection over the 1 Gigabit Ethernet interface on <code>SFP Port 0</code>.<br />
<br />
When streaming via SFP Port 0 at 1 Gb speeds, it is important that the connection is direct between the Host and USRP. Placing a switch or other network gear between the Host and USRP can reduce throughput of the transport link. It is also generally recommended to avoid using USB to Ethernet Adapters for the high speed streaming interface, as they may limit performance or cause periodic flow control errors. <br />
<br />
NOTE: The <code>HG</code> FPGA image must be loaded for <code>SFP Port 0</code> to operate at 1Gb speeds. If the <code>XG</code> image is loaded, the port will be unresponsive at 1Gb speeds. <br />
<br />
1. Configure your Host's Ethernet adapter as shown below. This interface should be separate from the 1Gb NIC/network which is connected to the 1Gb RJ45 management interface.<br />
<br />
IP Address: 192.168.10.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 1500<br />
<br />
NOTE: When operating <code>SFP Port 0</code> at 1Gb speeds, it is important to set a MTU of <code>1500</code> and not a value of <code>automatic</code>.<br />
<br />
2. Insert the RJ45 – SFP+ adapter into <code>SFP Port 0</code> .<br />
<br />
3. Connect the adapter to a host computer using the Ethernet cable to SFP0.<br />
<br />
The Green LED above <code>SFP Port 0</code> should illuminate.<br />
<br />
4. To test the connection, <code>ping</code> the device at address <code>192.168.10.2</code> from the host, as shown<br />
below:<br />
<br />
$ ping 192.168.10.2<br />
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.<br />
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=1.06 ms<br />
^C<br />
--- 192.168.10.2 ping statistics ---<br />
1 packets transmitted, 1 received, 0% packet loss, time 0ms<br />
rtt min/avg/max/mdev = 1.065/1.065/1.065/0.000 ms<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
===10Gb Streaming SFP Port 1===<br />
Complete the steps below to set up a streaming connection over the 10 Gigabit Ethernet interface on <code>SFP Port 1</code>.<br />
<br />
NOTE: Both the <code>HG</code> and <code>XG</code> FPGA images support 10Gb speeds over SFP Port 1. <br />
<br />
1. Configure your Host's 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.20.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
NOTE: When operating at 10Gb speeds, it is important to set a MTU of <code>9000</code> and not a value of <code>automatic</code>.<br />
<br />
2. Connect the USRP to a host computer using either a 10Gb SFP or Fiber cable to <code>SFP Port 1</code>.<br />
<br />
The Green LED above <code>SFP Port 1</code> should illuminate.<br />
<br />
3. To test the connection, <code>ping</code> the device at address <code>192.168.20.2</code> from the host, as shown<br />
below:<br />
<br />
$ ping 192.168.20.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
===Dual 10Gb Streaming SFP Ports 0/1===<br />
Complete the steps below to set up a streaming connections over the Dual 10 Gigabit Ethernet interface on <code>SFP Ports 0/1</code>.<br />
<br />
NOTE: The <code>XG</code> FPGA image must be loaded for <code>SFP Port 0</code> to operate at 10 Gb speeds. If the <code>HG</code> image is loaded, the port will be unresponsive at 10 Gb speeds. <br />
<br />
1. Configure your Host's #1 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.10.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
2. Configure your Host's #2 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.20.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
NOTE: When operating at 10Gb speeds, it is important to set a MTU of <code>9000</code> and not a value of <code>automatic</code>.<br />
<br />
3. Connect the USRP to a host computer using either a 10Gb SFP or Fiber cables to <code>SFP Ports 0/1</code>.<br />
<br />
The Green LEDs above <code>SFP Ports 0/1</code> should illuminate.<br />
<br />
4. To test the <code>SFP Port 0</code> connection, <code>ping</code> the device at address <code>192.168.10.2</code> from the host, as shown below:<br />
<br />
$ ping 192.168.10.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
5. To test the <code>SFP Port 1</code> connection, <code>ping</code> the device at address <code>192.168.20.2</code> from the host, as shown below:<br />
<br />
$ ping 192.168.20.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
For more details on Network Setup and Configuration, please see the “Interfaces and Connectivity” section on the [[N300/N310]] or [[N320/N321]] hardware resources pages.<br />
<br />
==Verifying Device Operation==<br />
Once you have successfully setup a management interface and streaming interface, you can now verify the devices operation using the included UHD utilities.<br />
<br />
===Subdevice Specification Mapping===<br />
====N300====<br />
The USRP N300 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = A:1<br />
<br />
====N310====<br />
The USRP N310 contains 4 channels, each represented on the front panel as <code>RF0-3</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
=====UHD 3.11.x.x - 3.12.x.x=====<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
* RF2 = C:0<br />
* RF3 = D:0<br />
<br />
=====UHD 3.13.x.x+=====<br />
* RF0 = A:0<br />
* RF1 = A:1<br />
* RF2 = B:0<br />
* RF3 = B:1<br />
<br />
====N320====<br />
The USRP N320 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
<br />
====N321====<br />
The USRP N321 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
<br />
Additional details of UHD Subdevice Specifications can be found here in the UHD Manual: http://files.ettus.com/manual/page_configuration.html#config_subdev<br />
<br />
===Supported Sample Rates===<br />
<br />
The USRP N300/N310 supports the three fixed Master Clock Rates listed below. <br />
<br />
* 122.88 MHz<br />
* 125.00 MHz<br />
* 153.60 MHz<br />
<br />
The USRP N320/N321 supports the three fixed Master Clock Rates listed below. <br />
<br />
* 200.00 MHz<br />
* 245.76 MHz<br />
* 250.00 MHz<br />
<br />
Sample rates as delivered to/from the host computer for USRP devices are constrained to follow several important rules.<br />
<br />
It is important to understand that strictly-integer decimation and interpolation are used within USRP hardware to meet the requested sample rate requirements of the application at hand. That means that the desired sample rate must meet the requirement that master-clock-rate/desired-sample-rate be an integer ratio. Further, it is strongly desirable for that ratio to be even. This ratio is the decimation (down-conversion) or interpolation (up-conversion) factor. The decimation or interpolation factor may be between 1 and 1024. There are further constraints on the decimation or interpolation factor. If the decimation or interpolation factor exceeds 128, then it must be evenly divisible by 2. If the decimation or interpolation factor exceeds 256, then it must be evenly divisible by 4.<br />
<br />
====Example Sample Rates====<br />
Listed below are common sample rates for the given master clock rates. This is not a complete listing of the supported sample rates.<br />
<br />
{| class="wikitable"<br />
!Master Clock Rate<br />
!colspan="20"|Decimation / Interpolation Rate <br> Host Sample Rate [Msps]<br />
|-<br />
<br />
|style="text-align:center;"| 1<br />
|style="text-align:center;"| 2<br />
|style="text-align:center;"| 4<br />
|style="text-align:center;"| 6<br />
|style="text-align:center;"| 8<br />
|style="text-align:center;"| 10<br />
|style="text-align:center;"| 12<br />
|style="text-align:center;"| 14<br />
|style="text-align:center;"| 16<br />
|style="text-align:center;"| 18<br />
|style="text-align:center;"| 20<br />
|style="text-align:center;"| 30<br />
|style="text-align:center;"| 32<br />
|style="text-align:center;"| 64<br />
|style="text-align:center;"| 100<br />
|style="text-align:center;"| 128<br />
|style="text-align:center;"| 200<br />
|style="text-align:center;"| 256<br />
|style="text-align:center;"| 512<br />
|style="text-align:center;"| 1024<br />
|-<br />
<br />
|style="text-align:center;"| 122.88e6<br />
|style="text-align:center;"| 61.44e6<br />
|style="text-align:center;"| 30.72e6<br />
|style="text-align:center;"| 20.48e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.288e6<br />
|style="text-align:center;"| 10.24e6<br />
|style="text-align:center;"| 8.7771e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 6.8267e6<br />
|style="text-align:center;"| 6.144e6<br />
|style="text-align:center;"| 4.096e6<br />
|style="text-align:center;"| 3.84e6<br />
|style="text-align:center;"| 1.92e6<br />
|style="text-align:center;"| 1.2288e6<br />
|style="text-align:center;"| 960e3<br />
|style="text-align:center;"| 614.4e3<br />
|style="text-align:center;"| 480e3<br />
|style="text-align:center;"| 240e3<br />
|style="text-align:center;"| 120e3<br />
|-<br />
<br />
|style="text-align:center;"| 125e6<br />
|style="text-align:center;"| 62.5e6<br />
|style="text-align:center;"| 31.25e6<br />
|style="text-align:center;"| 20.833e6<br />
|style="text-align:center;"| 15.625e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 10.417e6<br />
|style="text-align:center;"| 8.9286e6<br />
|style="text-align:center;"| 7.8125e6<br />
|style="text-align:center;"| 6.9444e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 4.1667e6<br />
|style="text-align:center;"| 3.90625e6<br />
|style="text-align:center;"| 1.953125e6<br />
|style="text-align:center;"| 1.25e6<br />
|style="text-align:center;"| 976.5625e3<br />
|style="text-align:center;"| 625e3<br />
|style="text-align:center;"| 488.28125e3<br />
|style="text-align:center;"| 244.14e3<br />
|style="text-align:center;"| 122.07e3<br />
<br />
|-<br />
<br />
|style="text-align:center;"| 153.6e6<br />
|style="text-align:center;"| 76.8e6<br />
|style="text-align:center;"| 38.4e6<br />
|style="text-align:center;"| 25.6e6<br />
|style="text-align:center;"| 19.2e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.8e6<br />
|style="text-align:center;"| 10.971e6<br />
|style="text-align:center;"| 9.6e6<br />
|style="text-align:center;"| 8.5333e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 5.12e6<br />
|style="text-align:center;"| 4.8e6<br />
|style="text-align:center;"| 2.4e6<br />
|style="text-align:center;"| 1.536e6<br />
|style="text-align:center;"| 1.2e6<br />
|style="text-align:center;"| 768e3<br />
|style="text-align:center;"| 600e3<br />
|style="text-align:center;"| 300e3<br />
|style="text-align:center;"| 150e3<br />
<br />
|-<br />
<br />
|}<br />
<br />
<br />
====N320/N321 Example Sample Rates====<br />
Listed below are common sample rates for the given master clock rates. This is not a complete listing of the supported sample rates.<br />
<br />
{| class="wikitable"<br />
!Master Clock Rate<br />
!colspan="20"|Decimation / Interpolation Rate <br> Host Sample Rate [Msps]<br />
|-<br />
<br />
|style="text-align:center;"| 1<br />
|style="text-align:center;"| 2<br />
|style="text-align:center;"| 4<br />
|style="text-align:center;"| 6<br />
|style="text-align:center;"| 8<br />
|style="text-align:center;"| 10<br />
|style="text-align:center;"| 12<br />
|style="text-align:center;"| 14<br />
|style="text-align:center;"| 16<br />
|style="text-align:center;"| 18<br />
|style="text-align:center;"| 20<br />
|style="text-align:center;"| 30<br />
|style="text-align:center;"| 32<br />
|style="text-align:center;"| 64<br />
|style="text-align:center;"| 100<br />
|style="text-align:center;"| 128<br />
|style="text-align:center;"| 200<br />
|style="text-align:center;"| 256<br />
|style="text-align:center;"| 512<br />
|style="text-align:center;"| 1024<br />
|-<br />
<br />
|style="text-align:center;"| 200e6<br />
|style="text-align:center;"| 100e6<br />
|style="text-align:center;"| 50e6<br />
|style="text-align:center;"| 33.33e6<br />
|style="text-align:center;"| 25e6<br />
|style="text-align:center;"| 20e6<br />
|style="text-align:center;"| 16.66e6<br />
|style="text-align:center;"| 14.2857e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 11.11e6<br />
|style="text-align:center;"| 10e6<br />
|style="text-align:center;"| 6.667e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 3.125e6<br />
|style="text-align:center;"| 2e6<br />
|style="text-align:center;"| 1.5625e6<br />
|style="text-align:center;"| 1e6<br />
|style="text-align:center;"| 781.25e3<br />
|style="text-align:center;"| 390.625e3<br />
|style="text-align:center;"| 195.3125e3<br />
<br />
|-<br />
<br />
|style="text-align:center;"| 245.76e6<br />
|style="text-align:center;"| 122.88e6<br />
|style="text-align:center;"| 61.44e6<br />
|style="text-align:center;"| 30.72e6<br />
|style="text-align:center;"| 20.48e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.288e6<br />
|style="text-align:center;"| 10.24e6<br />
|style="text-align:center;"| 8.7771e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 6.8267e6<br />
|style="text-align:center;"| 6.144e6<br />
|style="text-align:center;"| 4.096e6<br />
|style="text-align:center;"| 3.84e6<br />
|style="text-align:center;"| 1.92e6<br />
|style="text-align:center;"| 1.2288e6<br />
|style="text-align:center;"| 960e3<br />
|style="text-align:center;"| 614.4e3<br />
|style="text-align:center;"| 480e3<br />
|style="text-align:center;"| 240e3<br />
|-<br />
<br />
<br />
|style="text-align:center;"| 250e6<br />
|style="text-align:center;"| 125e6<br />
|style="text-align:center;"| 62.5e6<br />
|style="text-align:center;"| 31.25e6<br />
|style="text-align:center;"| 20.833e6<br />
|style="text-align:center;"| 15.625e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 10.417e6<br />
|style="text-align:center;"| 8.9286e6<br />
|style="text-align:center;"| 7.8125e6<br />
|style="text-align:center;"| 6.9444e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 4.1667e6<br />
|style="text-align:center;"| 3.90625e6<br />
|style="text-align:center;"| 1.953125e6<br />
|style="text-align:center;"| 1.25e6<br />
|style="text-align:center;"| 976.5625e3<br />
|style="text-align:center;"| 625e3<br />
|style="text-align:center;"| 488.28125e3<br />
|style="text-align:center;"| 244.14e3<br />
<br />
|-<br />
<br />
<br />
<br />
|}<br />
<br />
Additional information on Sample Rates can be found here in the UHD Manual: http://files.ettus.com/manual/page_general.html#general_sampleratenotes<br />
<br />
===Probe the USRP===<br />
<br />
====N300/N310====<br />
The UHD utility <code>uhd_usrp_probe</code> provides detailed information of the USRP device.<br />
<br />
From your host computer, run the command <code>uhd_usrp_probe</code>:<br />
<br />
<pre><br />
$ uhd_usrp_probe <br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.13.1.HEAD-0-ga0a71d10<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.10.2,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.13.1.0-gd3b7e90a<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal'.<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1355 MB/s)<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=192.168.10.2,clock_source=internal,time_source=internal,product=n310'.<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1358 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1355 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1345 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-313ABDA<br />
| | eeprom_version: 1<br />
| | mpm_version: 3.13.1.0-gd3b7e90a<br />
| | pid: 16962<br />
| | product: n310<br />
| | rev: 3<br />
| | rpc_connection: remote<br />
| | serial: 313ABDA<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.2<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_tpv, ref_locked, gps_time, gps_locked, temp, gps_sky, fan<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: AD9371 Dual ADC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: AD9371 Dual ADC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: AD9371 Dual DAC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: AD9371 Dual DAC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * DmaFIFO_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
<br />
<br />
<br />
</pre><br />
<br />
====N320====<br />
<br />
<pre><br />
<br />
$ uhd_usrp_probe <br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106600; UHD_3.14.0.0-0-g6875d061<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=127.0.0.1,type=n3xx,product=n320,serial=3181FFA,claimed=False<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.14.0.0-g6875d061<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 3181FFA<br />
[INFO] [MPM.Rhodium-0] Successfully loaded all peripherals!<br />
[INFO] [MPM.Rhodium-1] Successfully loaded all peripherals!<br />
[INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager] No QSFP board detected: Assuming it is disabled in the device tree overlay (e.g., HG, XG images).<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `time_source=internal,clock_source=internal'<br />
[INFO] [MPM.Rhodium-1] init() called with args `time_source=internal,clock_source=internal'<br />
[INFO] [MPM.Rhodium-0.init.LMK04828] LMK initialized and locked!<br />
[INFO] [MPM.Rhodium-1.init.LMK04828] LMK initialized and locked!<br />
[INFO] [MPM.Rhodium-1.DAC37J82] DAC PLL Locked!<br />
[INFO] [MPM.Rhodium-1.AD9695] ADC PLL Locked!<br />
[INFO] [MPM.Rhodium-1.init] JESD204B Link Initialization & Training Complete<br />
[INFO] [MPM.Rhodium-0.DAC37J82] DAC PLL Locked!<br />
[INFO] [MPM.Rhodium-0.AD9695] ADC PLL Locked!<br />
[INFO] [MPM.Rhodium-0.init] JESD204B Link Initialization & Training Complete<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'<br />
[INFO] [MPM.Rhodium-1] init() called with args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'<br />
[INFO] [0/Replay_0] Initializing block control (NOC ID: 0x4E91A00000000004)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/FIFO_0] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
[INFO] [0/FIFO_1] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-3181FFA<br />
| | eeprom_version: 2<br />
| | mpm_version: 3.14.0.0-g6875d061<br />
| | pid: 16962<br />
| | product: n320<br />
| | rev: 6<br />
| | rpc_connection: local<br />
| | serial: 3181FFA<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.3<br />
| | FPGA git hash: 3de8954.clean<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_tpv, temp, gps_sky, fan, gps_time, gps_locked, ref_locked, gps_gpgga<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A79<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A67<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A79<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A67<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * Replay_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
| | | * FIFO_0<br />
| | | * FIFO_1<br />
<br />
<br />
</pre><br />
<br />
<br />
====N321====<br />
<br />
<pre><br />
$ uhd_usrp_probe<br />
[INFO] [UHD] linux; GNU C++ version 7.3.1 20180712 (Red Hat 7.3.1-6); Boost_106400; UHD_3.14.0.0-0-g6875d061<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.20.2,type=n3xx,product=n320,serial=3166646,claimed=False,addr=192.168.20.2<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'<br />
[INFO] [MPM.Rhodium-1] init() called with args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'<br />
[INFO] [0/Replay_0] Initializing block control (NOC ID: 0x4E91A00000000004)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/FIFO_0] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
[INFO] [0/FIFO_1] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-3166646<br />
| | eeprom_version: 2<br />
| | mpm_version: 3.14.0.0-g6875d061<br />
| | pid: 16962<br />
| | product: n320<br />
| | rev: 6<br />
| | rpc_connection: remote<br />
| | serial: 3166646<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.3<br />
| | FPGA git hash: 3de8954.clean<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_sky, gps_time, gps_gpgga, gps_locked, fan, gps_tpv, ref_locked, temp<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D814<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D810<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D814<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D810<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * Replay_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
| | | * FIFO_0<br />
| | | * FIFO_1<br />
<br />
</pre><br />
<br />
If you see warnings such as:<br />
<br />
[WARNING] [UDP] The recv buffer could not be resized sufficiently.<br />
<br />
You need to resize the socket buffers for your network interface card:<br />
<br />
sudo sysctl -w net.core.rmem_max=288000<br />
sudo sysctl -w net.core.wmem_max=288000<br />
sudo sysctl -w net.core.rmem_max=33554432<br />
<br />
===ASCII Art Example===<br />
The UHD driver includes several example programs, which may serve as test programs or the basis for your application program. The source code can be obtained from the UHD repository on github at: https://github.com/EttusResearch/uhd/tree/master/host/examples<br />
<br />
You can quickly verify the operation of your USRP N3xx by running the <code>rx_ascii_art_dft</code> UHD example program. <br />
<br />
The <code>rx_ascii_art_dft</code> utility is a simple console based, real-time FFT display tool. It is not graphical in nature, so it can be easily run over an SSH connection within a terminal window, and does not need any graphical capability, such as X Windows, to be installed. It can also be run over a serial console connection, although this is not recommended, as the formatting may not render correctly.<br />
<br />
You can run a simple test of the N3xx USRP by connecting an antenna and observing the spectrum of a commercial FM radio station in real-time, following the steps below:<br />
<br />
1. Attach an antenna to the <code>Ch0/RX2</code> antenna port of the N3xx.<br />
<br />
2. From your host computer, run the command:<br />
<br />
'''N300/N310'''<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
</pre><br />
<br />
'''N320/N321'''<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=250e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
</pre><br />
<br />
NOTE: Modify the command line argument <code>freq</code> above to specify a tuning frequency for a strong local FM radio station. You will also need to update the IP Address to match your device IP.<br />
<br />
3. You should see a real-time FFT display of 2.5 MHz of spectrum, centered at the specified tuning frequency.<br />
<br />
4. Type "<code>Q</code>" or <code>Ctrl-C</code> to stop the program and to return to the Linux command line.<br />
<br />
5. You can run with the <code>--help</code> argument to see a description of all available command-line options.<br />
<br />
Example Output:<br />
<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
<br />
Creating the usrp device with: master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2...<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.11.1.HEAD-0-gad6b0935<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.151,type=n3xx,product=n310,serial=313ABDA,claimed=False,master_clock_rate=125e6,addr=192.168.10.2<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.11.1.0-gunknown<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=192.168.1.151,product=n310,master_clock_rate=125e6'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1336 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1338 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1346 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1350 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_2] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_3] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_2] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_3] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_2] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_3] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
TX Channel: 2<br />
TX DSP: 0<br />
TX Dboard: C<br />
TX Subdev: Magnesium<br />
TX Channel: 3<br />
TX DSP: 0<br />
TX Dboard: D<br />
TX Subdev: Magnesium<br />
<br />
Setting RX Rate: 2.500000 Msps...<br />
Actual RX Rate: 2.500000 Msps...<br />
<br />
Setting RX Freq: 98.500000 MHz...<br />
Actual RX Freq: 98.500000 MHz...<br />
<br />
Setting RX Gain: 50.000000 dB...<br />
Actual RX Gain: 50.000000 dB...<br />
<br />
Checking RX: all_los: locked ...<br />
<br />
Done!<br />
</pre><br />
<br />
===Benchmarking your system===<br />
Included with the UHD driver example programs is a utility, <code>benchmark_rate</code> to benchmark the transport link of the system. <br />
<br />
A system's maximum performance is dependent upon many factors. <code>benchmark_rate</code> will exercise the transport link and CPU of the system. <br />
<br />
====1 Gb Interface====<br />
NOTE: This example requires the <code>HG</code> FPGA image to be loaded.<br />
<br />
'''N300/N310'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 3.84 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=122.88e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 3.84e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 3.84e6 \<br />
--tx_subdev "A:0"<br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 1.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 1.25e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 1.25e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
'''N320/N321'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 3.84 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=245.76e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 3.84e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 3.84e6 \<br />
--tx_subdev "A:0"<br />
<br />
When streaming samples over a 1 Gb transport link, the maximum accumulative rate for all channels is 25 MS/s with a <code>sc16</code> OTW format. To achieve higher streaming rates, it is recommended to use the 10 Gb interfaces.<br />
<br />
====10 Gb Interface SFP 1====<br />
NOTE: This example will work with either the <code>HG</code> or <code>XG</code> FPGA image.<br />
<br />
'''N300/N310'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 31.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 31.25e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 31.25e6 \<br />
--tx_subdev "A:0" <br />
<br />
'''N320/N321'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 31.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=250e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 31.25e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 31.25e6 \<br />
--tx_subdev "A:0" <br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 30.72 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=122.88e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 30.72e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 30.72e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
'''N320/N321'''<br />
<br />
This example will test two full-duplex streams at 30.72 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=245.76e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 30.72e6 \<br />
--rx_subdev "A:0 B:0" \<br />
--tx_rate 30.72e6 \<br />
--tx_subdev "A:0 B:0"<br />
<br />
====Dual 10 Gb Interface====<br />
NOTE: This example requires the <code>XG</code> FPGA image to be loaded.<br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 62.5 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 62.5e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 62.5e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
<br />
'''N320/N321'''<br />
<br />
This example will test two full-duplex streams at 62.5 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=250e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 62.5e6 \<br />
--rx_subdev "A:0 B:0" \<br />
--tx_rate 62.5e6 \<br />
--tx_subdev "A:0 B:0"<br />
<br />
==USRP N3xx Device Specific Operations==<br />
<br />
===White Rabbit Ethernet-Based Synchronization===<br />
* [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]]<br />
<br />
<br />
===N320/N321===<br />
* [[USRP N320/N321 LO Distribution]]<br />
* [[5G NR EVM Measurements with the USRP N320/N321]]<br />
<br />
<br />
===Turning the Device Off/On===<br />
To avoid damaging the file system and causing any corruption, do not turn the device off with the power button without first shutting down the system. Use this command to cleanly and properly shut the system down:<br />
<br />
shutdown -h now<br />
<br />
===Enable Auto Booting===<br />
Auto booting of the N3xx when power is applied can be configured by enabling the flag on the device's EEPROM with the following command:<br />
<br />
eeprom-set-flags 0x1<br />
<br />
===Default Password===<br />
The default user is <code>root</code> and the password is empty (no password).<br />
<br />
It is recommended to update the <code>root</code> password, which can be done with the command <code>passwd</code>:<br />
<br />
Example Output:<br />
<br />
root@ni-n3xx-serial:~# passwd<br />
Changing password for root<br />
New password: <br />
Re-enter new password: <br />
passwd: password changed.<br />
<br />
==Technical Support and Community Knowledge Base==<br />
Technical support for USRP hardware is available through email only. If the product arrived in a nonfunctional state or you require technical assistance, please contact [mailto:support@ettus.com support@ettus.com]. Please allow 24 to 48 hours for response by email, depending on holidays and weekends, although we are often able to reply more quickly than that.<br />
<br />
We also recommend that you subscribe to the community mailing lists. The mailing lists have a responsive and knowledgeable community of hundreds of developers and technical users who are located around the world. When you join the community, you will be connected to this group of people who can help you learn about SDR and respond to your technical and specific questions. Often your question can be answered quickly on the mailing lists. Each mailing list also provides an archive of all past conversations and discussions going back many years. Your question or problem may have already been addressed before, and a relevant or helpful solution may already exist in the archive.<br />
<br />
Discussions involving the USRP hardware and the UHD software itself are best addressed through the '''usrp-users''' mailing list at [http://usrp-users.ettus.com http://usrp-users.ettus.com].<br />
<br />
Discussions involving the use of [http://gnuradio.org/ GNU Radio] with USRP hardware and UHD software are best addressed through the '''discuss-gnuradio''' mailing list at [https://lists.gnu.org/mailman/listinfo/discussgnuradio https://lists.gnu.org/mailman/listinfo/discussgnuradio].<br />
<br />
Discussions involving the use of [http://openbts.org/ OpenBTS®] with USRP hardware and UHD software are best addressed through the '''openbts-discuss''' mailing list at [https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss].<br />
<br />
The support page on our website is located at [https://www.ettus.com/support https://www.ettus.com/support]. The Knowledge Base is located at [https://kb.ettus.com https://kb.ettus.com].<br />
<br />
==Legal Considerations==<br />
Every country has laws governing the transmission and reception of radio signals. Users are solely responsible for insuring they use their USRP system in compliance with all applicable laws and regulations. Before attempting to transmit and/or receive on any frequency, we recommend that you determine what licenses may be required and what restrictions may apply.<br />
<br />
*NOTE: This USRP product is a piece of test equipment.<br />
<br />
==Sales and Ordering Support==<br />
If you have any non-technical questions related to your order, then please contact us by email at [mailto:orders@ettus.com orders@ettus.com], or by phone at +14086106399 (Monday-Friday, 8 AM - 5 PM, Pacific Time). Please be sure to include your order number and the serial number of your USRP.<br />
<br />
==Terms and Conditions of Sale==<br />
Terms and conditions of sale can be accessed online at the following link: http://www.ettus.com/legal/terms-and-conditions-of-sale<br />
<br />
[[Category:Getting Started Guides]]<br />
[[Category:N300]]<br />
[[Category:N310]]<br />
<br />
[[Category:N320]]<br />
<br />
[[Category:N321]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=USRP_N300/N310/N320/N321_Getting_Started_Guide&diff=6024USRP N300/N310/N320/N321 Getting Started Guide2024-03-11T19:25:31Z<p>MichaelDickens: /* Mender Update Process */ USRP login commandline prompt is "#" not "$" && what happens on reboot after successful mender install</p>
<hr />
<div>==Kit Contents==<br />
===N300===<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N300<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n300 kit.png|450px|center]]<br />
|}<br />
===N310===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N310<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n310 kit.png|500px|center]]<br />
|}<br />
<br />
===N320===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N320<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
|[[File:n320 kit.png|500px|center]] <br />
|}<br />
<br />
===N321===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N321<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n321 kit.png|500px|center]] <br />
|}<br />
==Verify the Contents of Your Kit==<br />
Ensure that your kit contains all the items listed above. If any items are missing, please contact sales@ettus.com immediately.<br />
<br />
==You Will Need==<br />
* microSD Card Writer<br />
<br />
* For Network Mode: A host computer with an available 1 or 10 Gigabit Ethernet interface for sample streaming. In addition to the Ethernet interface used for sampling streaming, your host computer will require a separate 1 Gigabit Ethernet interface for command and control streaming.<br />
<br />
* For Stand-Alone Embedded Mode: A host computer with an available 1 Gigabit Ethernet port or a USB 2.0 port to remotely access the embedded Linux operating system running on ARM CPU.<br />
<br />
==Proper Care and Handling==<br />
All Ettus Research products are individually tested before shipment. The USRP is guaranteed to be functional at the time it is received by the customer. Improper use or handling of the USRP can cause the device to become non-functional. Take the following precautions to prevent damage to the unit.<br />
<br />
* Never allow metal objects to touch the circuit board while powered.<br />
* Always properly terminate the transmit port with an antenna or 50Ω load.<br />
* Always handle the board with proper anti-static methods.<br />
* Never allow the board to directly or indirectly come into contact with any voltage spikes.<br />
* Never allow any water or condensing moisture to come into contact with the device.<br />
* Always use caution with FPGA, firmware, or software modifications.<br />
<br />
{|<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |Never apply more than -15 dBm of power into any RF input.<br />
|-<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |Always use at least 30dB attenuation if operating in loopback configuration<br />
|-<br />
|}<br />
<br />
==Install and Setup the Software Tools on Your Host Computer==<br />
In order to use your Universal Software Radio Peripheral (USRP™), you must have the software tools correctly installed and configured on your host computer. A step-by-step guide for doing this is available at the Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux|Linux]], [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X|OS X]] and [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows|Windows]] Application Notes.<br />
<br />
To find the latest release of UHD, see the UHD repository at https://github.com/EttusResearch/uhd.<br />
<br />
The USRP N310 requires UHD version 3.11.0.0 or later. <br />
<br />
The USRP N300 requires UHD version 3.12.0.0 or later.<br />
<br />
The USRP N320/N321 requires UHD version 3.14.0.0 or later. <br />
<br />
White Rabbit Ethernet-Based Synchronization of the N3xx USRP requires UHD version 3.12.0.0 or later. For additional details on White Rabbit Ethernet-Based Synchronization, please see the application note, [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]].<br />
<br />
'''When you receive a brand-new device, it is strongly recommended that you download the most recent filesystem image from the Ettus Research website and write it to the SD card that comes with the unit. It is not recommended that you use the SD card from the factory as-is. Instructions on downloading the latest filesystem image and writing it to the SD card are listed below.'''<br />
<br />
'''Note that if you are operating the device in Network Mode, the version of UHD running on the host computer and the USRP N3xx must match.'''<br />
<br />
==Connecting the Device==<br />
===Interfaces Overview===<br />
Listed below are the interfaces to connect to the USRP N3xx. Each interface has specific functionality, limitations and purpose. <br />
<br />
'''Serial Console'''<br />
<br />
The Serial Console provides a low level interface to the device typically used for debugging.<br />
<br />
'''1 Gigabit RJ45 Connection'''<br />
<br />
The 1 Gigabit RJ45 Connection interfaces with the on-board ARM CPU. When operated in "Network mode", this interface can optionally be used for UHD management traffic. Regardless of the operation mode (Network vs Embedded) this interface can be used to connect to the ARM via SSH. By default, the 1Gb RJ45 connection is configured to use a DHCP assigned IP address.<br />
<br />
'''Dual SFP+ Connections'''<br />
<br />
The Dual SFP+ Connections support multiple configurations for streaming high-speed, low-latency data, depending upon the FPGA image which is loaded.<br />
<br />
'''QSFP+ Connection (N320/ N321 Only)'''<br />
<br />
The QSFP+ Connection supports 2 x 10Gb lanes for streaming high-speed, low-latency data, while the onboard SFP0 connection is used for White Rabbit Ethernet-Based Synchronization.<br />
<br />
===Setting up a Serial Console Connection===<br />
It is possible to gain shell access to the device using a serial terminal emulator via the Serial Console port. Most Linux, OSX, or other Unix based operating systems have a tool called <code>screen</code> which can be used for this purpose. <br />
<br />
If you do not have <code>screen</code> installed, it can be installed via your package manager. For Ubuntu/Debian based operating systems it can be installed with <code>apt</code> such as:<br />
<br />
sudo apt install screen<br />
<br />
The default Baud Rate for the Serial Console is: <code>115200</code><br />
<br />
The exact device node you should attach to depends on your operating system's driver and other USB devices that might already be connected. Modern Linux systems offer alternatives to simply trying device nodes; instead, the OS might have a directory of symlinks under <code>/dev/serial/by-id</code>:<br />
<br />
$ ls /dev/serial/by-id<br />
usb-Digilent_Digilent_USB_Device_25163511FE00-if00-port0<br />
usb-Digilent_Digilent_USB_Device_25163511FE00-if01-port0<br />
usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if00-port0<br />
usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if01-port0<br />
<br />
NOTE: Exact names depend on the host operating system version and may differ.<br />
<br />
Every N3XX series device connected to USB will by default show up as four different devices. The devices labeled <code>"USB_to_UART_Bridge_Controller"</code> are the devices that offer a serial prompt. The first (with the <code>if00</code> suffix) connects to the <code>ARM CPU</code>, whereas the second connects to the <code>STM32 Microcontroller</code>. <br />
<br />
If you have multiple N3xx Serial Consoles connected to a single host, you may have to empirically test nodes. <br />
<br />
Connecting to the ARM CPU can be performed with the command:<br />
<br />
$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if00-port0 115200<br />
<br />
Upon starting the USRP N3xx, boot messages will appear and rapidly update. Once the boot process successfully completes, a login prompt like the following should appear:<br />
<br />
OpenEmbedded test ni-n3xx-313ABDA ttyPS0<br />
<br />
ni-n3xx-313ABDA login: <br />
<br />
Enter the username: <code>root</code> <br />
<br />
By default, the <code>root</code> user's password is left blank. Press the <code>Enter</code> key when prompted for a password.<br />
<br />
You should now be presented with a shell prompt similar to the following:<br />
<br />
root@ni-n3xx-<motherboard serial #>:~#<br />
<br />
Using the default configuration, the serial console will show all kernel log messages (which are not available when using SSH), and give access to the boot loader (U-boot prompt). This can be used to debug kernel or boot-loader issues more efficiently than when logged in via SSH.<br />
<br />
====Connecting to the microcontroller====<br />
<br />
Using the Serial Console interface, it is possible to connect to the STM32 microcontroller with the command below. The STM32 controls the power sequencing and several other low level device operations.<br />
<br />
$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if01-port0 115200<br />
<br />
The STM32 interface provides a very simple prompt. The command <code>help</code> will list all available commands. A direct connection to the microcontroller can be used to hard-reset the device without physically accessing it (i.e., emulating a power button press) and other low-level diagnostics.<br />
<br />
===Connecting to the ARM via SSH===<br />
By default, the RJ45 1Gb management interface is configured to be assigned a DHCP IP address. <br />
<br />
If you have access to a network which provides a DHCP server (such as a common router's LAN), attach the RJ45 1Gb port to this network. Details vary by vendor, however, most router management interfaces will provide a list of attached devices to the LAN including their IP address.<br />
<br />
Without access to a router management interface, you can identify the IP address by connecting to the ARM CPU via Serial Console as detailed in the section above and running the command <code>ip a</code>:<br />
<br />
Example Output:<br />
<br />
<pre><br />
# ip a<br />
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000<br />
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br />
inet 127.0.0.1/8 scope host lo<br />
valid_lft forever preferred_lft forever<br />
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
inet 192.168.1.151/24 brd 192.168.1.255 scope global dynamic eth0<br />
valid_lft 42865sec preferred_lft 42865sec<br />
3: sfp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
inet 192.168.10.2/24 brd 192.168.10.255 scope global sfp0<br />
valid_lft forever preferred_lft forever<br />
4: sfp1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
</pre><br />
<br />
If you do not have access to a network with a DHCP server, you can create one using the Linux utility <code>dnsmasq</code>:<br />
<br />
$ sudo dnsmasq -i <ETHERNET_ADAPTER_NAME> --dhcp-range=192.168.1.151,192.168.1.254 --except-interface=lo --bind-dynamic --no-daemon<br />
<br />
NOTE: Modify the value <code><ETHERNET_ADAPTER_NAME></code> to match the interface you would like to create a DHCP server on.<br />
<br />
After the device has obtained an IP address, you can remotely log into it from a Linux or macOS system with SSH, as shown below:<br />
<br />
$ ssh root@192.168.1.151<br />
<br />
NOTE: The IP address may vary depending on your network setup.<br />
<br />
NOTE: The <code>root</code> password default password is empty/blank.<br />
<br />
On Microsoft Windows, the SSH connection can be established using the third-party program Putty. <br />
<br />
After logging in, you should be presented with a shell like the following:<br />
<br />
root@ni-n3xx-<motherboard serial #>:~#<br />
<br />
==Updating the Linux File System==<br />
Before operating the device, it is strongly recommended to update to the latest version of the Embedded Linux file system. If you are operating the device in Network Mode, the version of UHD running on the host machine and N3xx USRP must match. <br />
<br />
There is two ways to update the file system for the N3xx USRP: <br />
<br />
1. Mender<br />
<br />
2. Physically remove microSD card from device and write a new file system to the microSD card. <br />
<br />
===File System Partition Layout===<br />
The SD Card is divided into four partitions. There is two root file system partitions, a boot partition and a data partition. <br />
<br />
Any data you would like to preserve through Mender updates should be saved to the <code>data</code> partition, which is mounted at <code>/data</code>.<br />
<br />
===Updating the file system with Mender===<br />
Mender is third-party software that enables remote updating of the root file system without physically accessing the device (see also the Mender website https://mender.io). Mender can be executed locally on the device, or a Mender server can be set up which can be used to remotely update an arbitrary number of USRP devices. Users can host their own local Mender server, or use servers hosted by Mender as a paid service; contact Mender for more information. <br />
<br />
====Mender Update Process====<br />
<br />
When updating the file system using Mender, the tool will overwrite the root file system partition that is not currently mounted. Any data stored in the root partitions will be permanently lost with a Mender update.<br />
<br />
After updating a partition with Mender, it will reboot into the newly updated partition. Only if the update is confirmed by the user, the update will be made permanent. This means that if an update fails, the device will be always able to reboot into the partition from which the update was originally launched, which presumably is in a working state. Another update can be launched now to correct the previous, failed update, until it works.<br />
<br />
To obtain the file system Mender image (these are files with a <code>.mender</code> suffix), run the following command on the host computer with Internet access:<br />
$ sudo uhd_images_downloader -t mender -t n3xx --yes<br />
<br />
Example Output:<br />
[INFO] Using base URL: https://files.ettus.com/binaries/cache/<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
365684 kB / 365684 kB (100%) n3xx_common_mender_default-v4.6.0.0.zip<br />
[INFO] Images download complete.<br />
<br />
The downloaded "zip" archive is extracted into the <code>Images destination</code> directory with the filename <code>usrp_n3xx_fs.mender</code>. Next, you will need to copy this Mender file system image from the <code>Images destination</code> directory to the USRP N3xx to have its filesystem changed. This can be done with the Linux utility <code>scp</code>, for example as follows:<br />
<br />
$ scp /usr/local/share/uhd/images/usrp_n3xx_fs.mender root@192.168.1.51:~/. <br />
<br />
Note: The path and IP may be different for your configuration; the command above assumes you're using the default UHD installation path of <code>/usr/local</code> and that the N3xx's IP is <code>192.168.1.51</code>.<br />
<br />
After copying the Mender file system image to the N3xx, connect to the N3xx to gain shell access via either the Serial Console or SSH.<br />
<br />
On the N3xx, you first need to determine the version of UHD currently running on the USRP; an easy way to do this is via the command<br />
# uhd_config_info --version<br />
<br />
Example output:<br />
UHD 3.14.1.1-0-g0347a6d8<br />
<br />
The mender command to execute is different for UHD version 4.0 or newer versus prior to version 4.0. For the former use <code>mender install</code> followed by the mender file; for the latter use <code>mender -f -rootfs</code> followed by the mender file. Starting with UHD version 4.0 one can use mender to upgrade or downgrade the UHD filesystem version between any UHD v4 versions (e.g., 4.1 to 4.6; 4.6 to 4.1). The following commands assume that the UHD filesystem is version 4; if not then substitute the other mender command.<br />
<br />
Run <code>mender install /path/to/latest.mender</code> to update the file system, e.g.:<br />
<br />
# mender install usrp_n3xx_fs.mender<br />
<br />
The artifact can also be stored on a remote server:<br />
# mender install <nowiki>http://server.name/path/to/latest.mender</nowiki><br />
<br />
This procedure will take a few minutes to complete. After mender has logged a successful update, reboot the device:<br />
# reboot<br />
<br />
Upon reboot log back in to the USRP N3xx and run the command <code>uhd_find_devices</code> to verify that the UHD version is as desired and that the command runs successfully -- it should find at a minimum the USRP it is being executed on and will find more if other USRPs are on the same network.<br />
<br />
If upon reboot the device is not working or the UHD version is not as desired, then the easiest way forward is to [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide#Updating_the_files_system_by_writing_the_disk_image overwrite the sdcard filesystem manually] with the desired UHD version.<br />
<br />
If upon reboot everything is as desired and the device seems functional, then commit the changes so that the boot loader knows to permanently boot into this partition:<br />
# mender commit<br />
<br />
To identify the currently installed Mender artifact from the command line, the following file can be queried on the N3xx:<br />
# cat /etc/mender/artifact_info<br />
<br />
If you are using a Mender server, the updates can be initiated from a web dashboard. From there, you can start the updates without having to log into the device, and you can update groups of USRPs with a few clicks in a web GUI. The dashboard can also be used to inspect the state of USRPs. This is a simple way to update groups of rack-mounted USRPs with custom file systems.<br />
<br />
For more information on updating the filesystem, refer to the [https://files.ettus.com/manual/ UHD Manual].<br />
<br />
===Updating the files system by writing the disk image===<br />
Please see the separate application note, [[Writing the USRP File System Disk Image to a SD Card]], for step-by-step instructions on writing the file system image to the SD card.<br />
<br />
==Updating the Network Configurations==<br />
The USRP N3xx systemd network configuration files are located at: <code>/data/network/</code><br />
<br />
# ls /data/network/<br />
eth0.network int0.network sfp0.network sfp1.network<br />
<br />
or for older versions of the file system: <code>/etc/systemd/network/</code><br />
<br />
# ls /etc/systemd/network/<br />
eth0.network sfp0.network sfp1.network<br />
<br />
For details on configuration please refer to the [https://www.freedesktop.org/software/systemd/man/systemd.network.html systemd-networkd manual pages].<br />
<br />
The factory settings are as follows:<br />
<pre><br />
eth0 (DHCP):<br />
<br />
[Match]<br />
Name=eth0<br />
<br />
[Network]<br />
DHCP=ipv4<br />
<br />
[DHCP]<br />
UseHostname=false<br />
<br />
sfp0 (static):<br />
<br />
[Match]<br />
Name=sfp0<br />
<br />
[Network]<br />
Address=192.168.10.2/24<br />
<br />
[Link]<br />
MTUBytes=9000<br />
<br />
sfp1 (static):<br />
<br />
[Match]<br />
Name=sfp1<br />
<br />
[Network]<br />
Address=192.168.20.2/24<br />
<br />
[Link]<br />
MTUBytes=9000<br />
</pre><br />
<br />
Additional notes on networking:<br />
<br />
* Care needs to be taken when editing these files on the device, since <code>vi</code> / <code>vim</code> sometimes generates undo files (e.g. <code>/etc/systemd/network/sfp0.network~</code>), that <code>systemd-networkd</code> might accidentally pick up.<br />
* Temporarily setting the IP addresses or MTU sizes via <code>ifconfig</code> or other command line tools will only change the value until the next reboot or reload of the FPGA image.<br />
* If the MTU of the device and host computers differ, streaming issues can occur.<br />
* Streaming via SFP0 at 1 Gb rates requires a MTU of <code>1500</code><br />
* Streaming via SFP0 at 10 Gb rates requires a MTU of <code>9000</code><br />
<br />
For addition details on network configuration here: https://files.ettus.com/manual/page_usrp_n3xx.html#n3xx_network_configuration<br />
<br />
==Updating the FPGA Image==<br />
<br />
===Network Mode FPGA Image Update===<br />
The FPGA image should match the version of UHD installed on the host computer, when operated in Network mode. Connect the device to the host computer using either the RJ45 or SFP+ port, refer to the section above for detailed instructions. <br />
<br />
To obtain all the FPGA images for a specific version of UHD, run the following command on the host computer with internet access:<br />
<br />
$ sudo uhd_images_downloader<br />
<br />
Example Output:<br />
<br />
$ sudo uhd_images_downloader<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
00006 kB / 00006 kB (100%) usrp1_b100_fw_default-g6bea23d.zip<br />
19810 kB / 19810 kB (100%) x3xx_x310_fpga_default-gf1ba32fe.zip<br />
02757 kB / 02757 kB (100%) usrp2_n210_fpga_default-g6bea23d.zip<br />
02123 kB / 02123 kB (100%) n230_n230_fpga_default-ge57dfe0.zip<br />
00522 kB / 00522 kB (100%) usrp1_b100_fpga_default-g6bea23d.zip<br />
00491 kB / 00491 kB (100%) b2xx_b200_fpga_default-ge57dfe0.zip<br />
02415 kB / 02415 kB (100%) usrp2_n200_fpga_default-g6bea23d.zip<br />
08988 kB / 08988 kB (100%) e3xx_e320_fpga_default-g3de8954a.zip<br />
23045 kB / 23045 kB (100%) n3xx_n310_fpga_default-g3de8954a.zip<br />
00523 kB / 00523 kB (100%) b2xx_b205mini_fpga_default-ge57dfe0.zip<br />
18937 kB / 18937 kB (100%) x3xx_x300_fpga_default-gf1ba32fe.zip<br />
00017 kB / 00017 kB (100%) octoclock_octoclock_fw_default-g14000041.zip<br />
00007 kB / 00007 kB (100%) usrp2_usrp2_fw_default-g6bea23d.zip<br />
00009 kB / 00009 kB (100%) usrp2_n200_fw_default-g6bea23d.zip<br />
00450 kB / 00450 kB (100%) usrp2_usrp2_fpga_default-g6bea23d.zip<br />
00144 kB / 00144 kB (100%) b2xx_common_fw_default-ga69ab0c.zip<br />
25107 kB / 25107 kB (100%) n3xx_n320_fpga_default-g3de8954a.zip<br />
00464 kB / 00464 kB (100%) b2xx_b200mini_fpga_default-ge57dfe0.zip<br />
00319 kB / 00319 kB (100%) usrp1_usrp1_fpga_default-g6bea23d.zip<br />
04839 kB / 04839 kB (100%) usb_common_windrv_default-g14000041.zip<br />
00009 kB / 00009 kB (100%) usrp2_n210_fw_default-g6bea23d.zip<br />
16065 kB / 16065 kB (100%) n3xx_n300_fpga_default-g3de8954a.zip<br />
05578 kB / 05578 kB (100%) e3xx_e310_fpga_default-g4bc2c6f.zip<br />
00885 kB / 00885 kB (100%) b2xx_b210_fpga_default-ge57dfe0.zip<br />
[INFO] Images download complete.<br />
<br />
<br />
NOTE: In the above example output, the Images Destination folder is printed:<br />
<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
<br />
To list the N3xx FPGA images with a full path, run the command:<br />
<br />
$ ls -w 1 /usr/local/share/uhd/images/usrp_n3*.bit<br />
<br />
/usr/local/share/uhd/images/usrp_n300_fpga_AA.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_AA.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_AQ.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_XQ.bit<br />
<br />
To update the default <code>HG</code> variant of FPGA image, run the command:<br />
<br />
$ uhd_image_loader --args "type=n3xx,addr=<N3xx_IP_ADDR>,fpga=HG"<br />
<br />
Example Output:<br />
<br />
uhd_image_loader --args "type=n3xx,addr=192.168.1.151,fpga=HG"<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.11.1.HEAD-0-gad6b0935<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.151,type=n3xx,product=n310,serial=313ABDA,claimed=False,skip_init=1<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.11.1.0-gunknown<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPMD] Claimed device without full initialization.<br />
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.<br />
[INFO] [MPM.PeriphManager] Updating component `fpga'<br />
[INFO] [MPM.PeriphManager] Updating component `dts'<br />
[INFO] [MPM.RPCServer] Resetting peripheral manager.<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.<br />
<br />
<br />
To load a different default FPGA image (i.e. <code>XG</code>, <code>WG</code>), modify the device argument <code>fpga=</code> to a value of <code>fpga=XG</code> or <code>fpga=WG</code>.<br />
<br />
To specify the path to a custom FPGA image, use the <code>--fpga-path</code> argument. <br />
<br />
$ uhd_image_loader --args "type=n3xx,addr=<N3xx_IP_ADDR>" --fpga-path=/path/to/custom/fpga.bit<br />
<br />
The Verilog code for the FPGA in the USRP N3xx is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
<br />
===Embedded Mode FPGA Image Update===<br />
<br />
It is possible to update the FPGA image when operated in Embedded mode. Connect to the ARM CPU via Serial Console or SSH as detailed in the section above. <br />
<br />
Updating the FPGA image from the ARM CPU is the same as detailed above for a Network mode update, except it is not required to provide an <code>addr</code> device argument. <br />
<br />
uhd_image_loader --args "type=n3xx,fpga=HG"<br />
<br />
<pre><br />
root@ni-n3xx-313ABDA:~# uhd_image_loader --args "type=n3xx,fpga=HG"<br />
[INFO] [UHD] linux; GNU C++ version 7.2.0; Boost_106400; UHD_3.11.1.0-0-unknown<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=127.0.0.1,type=n3xx,product=n310,serial=313ABDA,claimed=False,skip_init=1<br />
[INFO] [MPMD] Claimed device without full initialization.<br />
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.<br />
[INFO] [MPM.PeriphManager] Updating component `fpga'<br />
[INFO] [MPM.PeriphManager] Updating component `dts'<br />
[INFO] [MPM.RPCServer] Resetting peripheral manager.<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.<br />
</pre><br />
<br />
For more information on updating the FPGA image, refer to the UHD Manual at http://uhd.ettus.com .<br />
<br />
==Setting Up a Streaming Connection==<br />
The device supports multiple, high-speed, low-latency interfaces on the SFP+ ports for streaming samples to the host computer. <br />
<br />
===1Gb Streaming SFP Port 0===<br />
Complete the steps below to set up a streaming connection over the 1 Gigabit Ethernet interface on <code>SFP Port 0</code>.<br />
<br />
When streaming via SFP Port 0 at 1 Gb speeds, it is important that the connection is direct between the Host and USRP. Placing a switch or other network gear between the Host and USRP can reduce throughput of the transport link. It is also generally recommended to avoid using USB to Ethernet Adapters for the high speed streaming interface, as they may limit performance or cause periodic flow control errors. <br />
<br />
NOTE: The <code>HG</code> FPGA image must be loaded for <code>SFP Port 0</code> to operate at 1Gb speeds. If the <code>XG</code> image is loaded, the port will be unresponsive at 1Gb speeds. <br />
<br />
1. Configure your Host's Ethernet adapter as shown below. This interface should be separate from the 1Gb NIC/network which is connected to the 1Gb RJ45 management interface.<br />
<br />
IP Address: 192.168.10.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 1500<br />
<br />
NOTE: When operating <code>SFP Port 0</code> at 1Gb speeds, it is important to set a MTU of <code>1500</code> and not a value of <code>automatic</code>.<br />
<br />
2. Insert the RJ45 – SFP+ adapter into <code>SFP Port 0</code> .<br />
<br />
3. Connect the adapter to a host computer using the Ethernet cable to SFP0.<br />
<br />
The Green LED above <code>SFP Port 0</code> should illuminate.<br />
<br />
4. To test the connection, <code>ping</code> the device at address <code>192.168.10.2</code> from the host, as shown<br />
below:<br />
<br />
$ ping 192.168.10.2<br />
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.<br />
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=1.06 ms<br />
^C<br />
--- 192.168.10.2 ping statistics ---<br />
1 packets transmitted, 1 received, 0% packet loss, time 0ms<br />
rtt min/avg/max/mdev = 1.065/1.065/1.065/0.000 ms<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
===10Gb Streaming SFP Port 1===<br />
Complete the steps below to set up a streaming connection over the 10 Gigabit Ethernet interface on <code>SFP Port 1</code>.<br />
<br />
NOTE: Both the <code>HG</code> and <code>XG</code> FPGA images support 10Gb speeds over SFP Port 1. <br />
<br />
1. Configure your Host's 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.20.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
NOTE: When operating at 10Gb speeds, it is important to set a MTU of <code>9000</code> and not a value of <code>automatic</code>.<br />
<br />
2. Connect the USRP to a host computer using either a 10Gb SFP or Fiber cable to <code>SFP Port 1</code>.<br />
<br />
The Green LED above <code>SFP Port 1</code> should illuminate.<br />
<br />
3. To test the connection, <code>ping</code> the device at address <code>192.168.20.2</code> from the host, as shown<br />
below:<br />
<br />
$ ping 192.168.20.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
===Dual 10Gb Streaming SFP Ports 0/1===<br />
Complete the steps below to set up a streaming connections over the Dual 10 Gigabit Ethernet interface on <code>SFP Ports 0/1</code>.<br />
<br />
NOTE: The <code>XG</code> FPGA image must be loaded for <code>SFP Port 0</code> to operate at 10 Gb speeds. If the <code>HG</code> image is loaded, the port will be unresponsive at 10 Gb speeds. <br />
<br />
1. Configure your Host's #1 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.10.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
2. Configure your Host's #2 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.20.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
NOTE: When operating at 10Gb speeds, it is important to set a MTU of <code>9000</code> and not a value of <code>automatic</code>.<br />
<br />
3. Connect the USRP to a host computer using either a 10Gb SFP or Fiber cables to <code>SFP Ports 0/1</code>.<br />
<br />
The Green LEDs above <code>SFP Ports 0/1</code> should illuminate.<br />
<br />
4. To test the <code>SFP Port 0</code> connection, <code>ping</code> the device at address <code>192.168.10.2</code> from the host, as shown below:<br />
<br />
$ ping 192.168.10.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
5. To test the <code>SFP Port 1</code> connection, <code>ping</code> the device at address <code>192.168.20.2</code> from the host, as shown below:<br />
<br />
$ ping 192.168.20.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
For more details on Network Setup and Configuration, please see the “Interfaces and Connectivity” section on the [[N300/N310]] or [[N320/N321]] hardware resources pages.<br />
<br />
==Verifying Device Operation==<br />
Once you have successfully setup a management interface and streaming interface, you can now verify the devices operation using the included UHD utilities.<br />
<br />
===Subdevice Specification Mapping===<br />
====N300====<br />
The USRP N300 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = A:1<br />
<br />
====N310====<br />
The USRP N310 contains 4 channels, each represented on the front panel as <code>RF0-3</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
=====UHD 3.11.x.x - 3.12.x.x=====<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
* RF2 = C:0<br />
* RF3 = D:0<br />
<br />
=====UHD 3.13.x.x+=====<br />
* RF0 = A:0<br />
* RF1 = A:1<br />
* RF2 = B:0<br />
* RF3 = B:1<br />
<br />
====N320====<br />
The USRP N320 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
<br />
====N321====<br />
The USRP N321 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
<br />
Additional details of UHD Subdevice Specifications can be found here in the UHD Manual: http://files.ettus.com/manual/page_configuration.html#config_subdev<br />
<br />
===Supported Sample Rates===<br />
<br />
The USRP N300/N310 supports the three fixed Master Clock Rates listed below. <br />
<br />
* 122.88 MHz<br />
* 125.00 MHz<br />
* 153.60 MHz<br />
<br />
The USRP N320/N321 supports the three fixed Master Clock Rates listed below. <br />
<br />
* 200.00 MHz<br />
* 245.76 MHz<br />
* 250.00 MHz<br />
<br />
Sample rates as delivered to/from the host computer for USRP devices are constrained to follow several important rules.<br />
<br />
It is important to understand that strictly-integer decimation and interpolation are used within USRP hardware to meet the requested sample rate requirements of the application at hand. That means that the desired sample rate must meet the requirement that master-clock-rate/desired-sample-rate be an integer ratio. Further, it is strongly desirable for that ratio to be even. This ratio is the decimation (down-conversion) or interpolation (up-conversion) factor. The decimation or interpolation factor may be between 1 and 1024. There are further constraints on the decimation or interpolation factor. If the decimation or interpolation factor exceeds 128, then it must be evenly divisible by 2. If the decimation or interpolation factor exceeds 256, then it must be evenly divisible by 4.<br />
<br />
====Example Sample Rates====<br />
Listed below are common sample rates for the given master clock rates. This is not a complete listing of the supported sample rates.<br />
<br />
{| class="wikitable"<br />
!Master Clock Rate<br />
!colspan="20"|Decimation / Interpolation Rate <br> Host Sample Rate [Msps]<br />
|-<br />
<br />
|style="text-align:center;"| 1<br />
|style="text-align:center;"| 2<br />
|style="text-align:center;"| 4<br />
|style="text-align:center;"| 6<br />
|style="text-align:center;"| 8<br />
|style="text-align:center;"| 10<br />
|style="text-align:center;"| 12<br />
|style="text-align:center;"| 14<br />
|style="text-align:center;"| 16<br />
|style="text-align:center;"| 18<br />
|style="text-align:center;"| 20<br />
|style="text-align:center;"| 30<br />
|style="text-align:center;"| 32<br />
|style="text-align:center;"| 64<br />
|style="text-align:center;"| 100<br />
|style="text-align:center;"| 128<br />
|style="text-align:center;"| 200<br />
|style="text-align:center;"| 256<br />
|style="text-align:center;"| 512<br />
|style="text-align:center;"| 1024<br />
|-<br />
<br />
|style="text-align:center;"| 122.88e6<br />
|style="text-align:center;"| 61.44e6<br />
|style="text-align:center;"| 30.72e6<br />
|style="text-align:center;"| 20.48e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.288e6<br />
|style="text-align:center;"| 10.24e6<br />
|style="text-align:center;"| 8.7771e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 6.8267e6<br />
|style="text-align:center;"| 6.144e6<br />
|style="text-align:center;"| 4.096e6<br />
|style="text-align:center;"| 3.84e6<br />
|style="text-align:center;"| 1.92e6<br />
|style="text-align:center;"| 1.2288e6<br />
|style="text-align:center;"| 960e3<br />
|style="text-align:center;"| 614.4e3<br />
|style="text-align:center;"| 480e3<br />
|style="text-align:center;"| 240e3<br />
|style="text-align:center;"| 120e3<br />
|-<br />
<br />
|style="text-align:center;"| 125e6<br />
|style="text-align:center;"| 62.5e6<br />
|style="text-align:center;"| 31.25e6<br />
|style="text-align:center;"| 20.833e6<br />
|style="text-align:center;"| 15.625e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 10.417e6<br />
|style="text-align:center;"| 8.9286e6<br />
|style="text-align:center;"| 7.8125e6<br />
|style="text-align:center;"| 6.9444e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 4.1667e6<br />
|style="text-align:center;"| 3.90625e6<br />
|style="text-align:center;"| 1.953125e6<br />
|style="text-align:center;"| 1.25e6<br />
|style="text-align:center;"| 976.5625e3<br />
|style="text-align:center;"| 625e3<br />
|style="text-align:center;"| 488.28125e3<br />
|style="text-align:center;"| 244.14e3<br />
|style="text-align:center;"| 122.07e3<br />
<br />
|-<br />
<br />
|style="text-align:center;"| 153.6e6<br />
|style="text-align:center;"| 76.8e6<br />
|style="text-align:center;"| 38.4e6<br />
|style="text-align:center;"| 25.6e6<br />
|style="text-align:center;"| 19.2e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.8e6<br />
|style="text-align:center;"| 10.971e6<br />
|style="text-align:center;"| 9.6e6<br />
|style="text-align:center;"| 8.5333e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 5.12e6<br />
|style="text-align:center;"| 4.8e6<br />
|style="text-align:center;"| 2.4e6<br />
|style="text-align:center;"| 1.536e6<br />
|style="text-align:center;"| 1.2e6<br />
|style="text-align:center;"| 768e3<br />
|style="text-align:center;"| 600e3<br />
|style="text-align:center;"| 300e3<br />
|style="text-align:center;"| 150e3<br />
<br />
|-<br />
<br />
|}<br />
<br />
<br />
====N320/N321 Example Sample Rates====<br />
Listed below are common sample rates for the given master clock rates. This is not a complete listing of the supported sample rates.<br />
<br />
{| class="wikitable"<br />
!Master Clock Rate<br />
!colspan="20"|Decimation / Interpolation Rate <br> Host Sample Rate [Msps]<br />
|-<br />
<br />
|style="text-align:center;"| 1<br />
|style="text-align:center;"| 2<br />
|style="text-align:center;"| 4<br />
|style="text-align:center;"| 6<br />
|style="text-align:center;"| 8<br />
|style="text-align:center;"| 10<br />
|style="text-align:center;"| 12<br />
|style="text-align:center;"| 14<br />
|style="text-align:center;"| 16<br />
|style="text-align:center;"| 18<br />
|style="text-align:center;"| 20<br />
|style="text-align:center;"| 30<br />
|style="text-align:center;"| 32<br />
|style="text-align:center;"| 64<br />
|style="text-align:center;"| 100<br />
|style="text-align:center;"| 128<br />
|style="text-align:center;"| 200<br />
|style="text-align:center;"| 256<br />
|style="text-align:center;"| 512<br />
|style="text-align:center;"| 1024<br />
|-<br />
<br />
|style="text-align:center;"| 200e6<br />
|style="text-align:center;"| 100e6<br />
|style="text-align:center;"| 50e6<br />
|style="text-align:center;"| 33.33e6<br />
|style="text-align:center;"| 25e6<br />
|style="text-align:center;"| 20e6<br />
|style="text-align:center;"| 16.66e6<br />
|style="text-align:center;"| 14.2857e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 11.11e6<br />
|style="text-align:center;"| 10e6<br />
|style="text-align:center;"| 6.667e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 3.125e6<br />
|style="text-align:center;"| 2e6<br />
|style="text-align:center;"| 1.5625e6<br />
|style="text-align:center;"| 1e6<br />
|style="text-align:center;"| 781.25e3<br />
|style="text-align:center;"| 390.625e3<br />
|style="text-align:center;"| 195.3125e3<br />
<br />
|-<br />
<br />
|style="text-align:center;"| 245.76e6<br />
|style="text-align:center;"| 122.88e6<br />
|style="text-align:center;"| 61.44e6<br />
|style="text-align:center;"| 30.72e6<br />
|style="text-align:center;"| 20.48e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.288e6<br />
|style="text-align:center;"| 10.24e6<br />
|style="text-align:center;"| 8.7771e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 6.8267e6<br />
|style="text-align:center;"| 6.144e6<br />
|style="text-align:center;"| 4.096e6<br />
|style="text-align:center;"| 3.84e6<br />
|style="text-align:center;"| 1.92e6<br />
|style="text-align:center;"| 1.2288e6<br />
|style="text-align:center;"| 960e3<br />
|style="text-align:center;"| 614.4e3<br />
|style="text-align:center;"| 480e3<br />
|style="text-align:center;"| 240e3<br />
|-<br />
<br />
<br />
|style="text-align:center;"| 250e6<br />
|style="text-align:center;"| 125e6<br />
|style="text-align:center;"| 62.5e6<br />
|style="text-align:center;"| 31.25e6<br />
|style="text-align:center;"| 20.833e6<br />
|style="text-align:center;"| 15.625e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 10.417e6<br />
|style="text-align:center;"| 8.9286e6<br />
|style="text-align:center;"| 7.8125e6<br />
|style="text-align:center;"| 6.9444e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 4.1667e6<br />
|style="text-align:center;"| 3.90625e6<br />
|style="text-align:center;"| 1.953125e6<br />
|style="text-align:center;"| 1.25e6<br />
|style="text-align:center;"| 976.5625e3<br />
|style="text-align:center;"| 625e3<br />
|style="text-align:center;"| 488.28125e3<br />
|style="text-align:center;"| 244.14e3<br />
<br />
|-<br />
<br />
<br />
<br />
|}<br />
<br />
Additional information on Sample Rates can be found here in the UHD Manual: http://files.ettus.com/manual/page_general.html#general_sampleratenotes<br />
<br />
===Probe the USRP===<br />
<br />
====N300/N310====<br />
The UHD utility <code>uhd_usrp_probe</code> provides detailed information of the USRP device.<br />
<br />
From your host computer, run the command <code>uhd_usrp_probe</code>:<br />
<br />
<pre><br />
$ uhd_usrp_probe <br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.13.1.HEAD-0-ga0a71d10<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.10.2,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.13.1.0-gd3b7e90a<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal'.<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1355 MB/s)<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=192.168.10.2,clock_source=internal,time_source=internal,product=n310'.<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1358 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1355 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1345 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-313ABDA<br />
| | eeprom_version: 1<br />
| | mpm_version: 3.13.1.0-gd3b7e90a<br />
| | pid: 16962<br />
| | product: n310<br />
| | rev: 3<br />
| | rpc_connection: remote<br />
| | serial: 313ABDA<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.2<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_tpv, ref_locked, gps_time, gps_locked, temp, gps_sky, fan<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: AD9371 Dual ADC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: AD9371 Dual ADC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: AD9371 Dual DAC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: AD9371 Dual DAC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * DmaFIFO_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
<br />
<br />
<br />
</pre><br />
<br />
====N320====<br />
<br />
<pre><br />
<br />
$ uhd_usrp_probe <br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106600; UHD_3.14.0.0-0-g6875d061<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=127.0.0.1,type=n3xx,product=n320,serial=3181FFA,claimed=False<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.14.0.0-g6875d061<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 3181FFA<br />
[INFO] [MPM.Rhodium-0] Successfully loaded all peripherals!<br />
[INFO] [MPM.Rhodium-1] Successfully loaded all peripherals!<br />
[INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager] No QSFP board detected: Assuming it is disabled in the device tree overlay (e.g., HG, XG images).<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `time_source=internal,clock_source=internal'<br />
[INFO] [MPM.Rhodium-1] init() called with args `time_source=internal,clock_source=internal'<br />
[INFO] [MPM.Rhodium-0.init.LMK04828] LMK initialized and locked!<br />
[INFO] [MPM.Rhodium-1.init.LMK04828] LMK initialized and locked!<br />
[INFO] [MPM.Rhodium-1.DAC37J82] DAC PLL Locked!<br />
[INFO] [MPM.Rhodium-1.AD9695] ADC PLL Locked!<br />
[INFO] [MPM.Rhodium-1.init] JESD204B Link Initialization & Training Complete<br />
[INFO] [MPM.Rhodium-0.DAC37J82] DAC PLL Locked!<br />
[INFO] [MPM.Rhodium-0.AD9695] ADC PLL Locked!<br />
[INFO] [MPM.Rhodium-0.init] JESD204B Link Initialization & Training Complete<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'<br />
[INFO] [MPM.Rhodium-1] init() called with args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'<br />
[INFO] [0/Replay_0] Initializing block control (NOC ID: 0x4E91A00000000004)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/FIFO_0] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
[INFO] [0/FIFO_1] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-3181FFA<br />
| | eeprom_version: 2<br />
| | mpm_version: 3.14.0.0-g6875d061<br />
| | pid: 16962<br />
| | product: n320<br />
| | rev: 6<br />
| | rpc_connection: local<br />
| | serial: 3181FFA<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.3<br />
| | FPGA git hash: 3de8954.clean<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_tpv, temp, gps_sky, fan, gps_time, gps_locked, ref_locked, gps_gpgga<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A79<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A67<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A79<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A67<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * Replay_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
| | | * FIFO_0<br />
| | | * FIFO_1<br />
<br />
<br />
</pre><br />
<br />
<br />
====N321====<br />
<br />
<pre><br />
$ uhd_usrp_probe<br />
[INFO] [UHD] linux; GNU C++ version 7.3.1 20180712 (Red Hat 7.3.1-6); Boost_106400; UHD_3.14.0.0-0-g6875d061<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.20.2,type=n3xx,product=n320,serial=3166646,claimed=False,addr=192.168.20.2<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'<br />
[INFO] [MPM.Rhodium-1] init() called with args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'<br />
[INFO] [0/Replay_0] Initializing block control (NOC ID: 0x4E91A00000000004)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/FIFO_0] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
[INFO] [0/FIFO_1] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-3166646<br />
| | eeprom_version: 2<br />
| | mpm_version: 3.14.0.0-g6875d061<br />
| | pid: 16962<br />
| | product: n320<br />
| | rev: 6<br />
| | rpc_connection: remote<br />
| | serial: 3166646<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.3<br />
| | FPGA git hash: 3de8954.clean<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_sky, gps_time, gps_gpgga, gps_locked, fan, gps_tpv, ref_locked, temp<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D814<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D810<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D814<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D810<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * Replay_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
| | | * FIFO_0<br />
| | | * FIFO_1<br />
<br />
</pre><br />
<br />
If you see warnings such as:<br />
<br />
[WARNING] [UDP] The recv buffer could not be resized sufficiently.<br />
<br />
You need to resize the socket buffers for your network interface card:<br />
<br />
sudo sysctl -w net.core.rmem_max=288000<br />
sudo sysctl -w net.core.wmem_max=288000<br />
sudo sysctl -w net.core.rmem_max=33554432<br />
<br />
===ASCII Art Example===<br />
The UHD driver includes several example programs, which may serve as test programs or the basis for your application program. The source code can be obtained from the UHD repository on github at: https://github.com/EttusResearch/uhd/tree/master/host/examples<br />
<br />
You can quickly verify the operation of your USRP N3xx by running the <code>rx_ascii_art_dft</code> UHD example program. <br />
<br />
The <code>rx_ascii_art_dft</code> utility is a simple console based, real-time FFT display tool. It is not graphical in nature, so it can be easily run over an SSH connection within a terminal window, and does not need any graphical capability, such as X Windows, to be installed. It can also be run over a serial console connection, although this is not recommended, as the formatting may not render correctly.<br />
<br />
You can run a simple test of the N3xx USRP by connecting an antenna and observing the spectrum of a commercial FM radio station in real-time, following the steps below:<br />
<br />
1. Attach an antenna to the <code>Ch0/RX2</code> antenna port of the N3xx.<br />
<br />
2. From your host computer, run the command:<br />
<br />
'''N300/N310'''<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
</pre><br />
<br />
'''N320/N321'''<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=250e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
</pre><br />
<br />
NOTE: Modify the command line argument <code>freq</code> above to specify a tuning frequency for a strong local FM radio station. You will also need to update the IP Address to match your device IP.<br />
<br />
3. You should see a real-time FFT display of 2.5 MHz of spectrum, centered at the specified tuning frequency.<br />
<br />
4. Type "<code>Q</code>" or <code>Ctrl-C</code> to stop the program and to return to the Linux command line.<br />
<br />
5. You can run with the <code>--help</code> argument to see a description of all available command-line options.<br />
<br />
Example Output:<br />
<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
<br />
Creating the usrp device with: master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2...<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.11.1.HEAD-0-gad6b0935<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.151,type=n3xx,product=n310,serial=313ABDA,claimed=False,master_clock_rate=125e6,addr=192.168.10.2<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.11.1.0-gunknown<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=192.168.1.151,product=n310,master_clock_rate=125e6'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1336 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1338 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1346 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1350 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_2] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_3] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_2] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_3] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_2] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_3] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
TX Channel: 2<br />
TX DSP: 0<br />
TX Dboard: C<br />
TX Subdev: Magnesium<br />
TX Channel: 3<br />
TX DSP: 0<br />
TX Dboard: D<br />
TX Subdev: Magnesium<br />
<br />
Setting RX Rate: 2.500000 Msps...<br />
Actual RX Rate: 2.500000 Msps...<br />
<br />
Setting RX Freq: 98.500000 MHz...<br />
Actual RX Freq: 98.500000 MHz...<br />
<br />
Setting RX Gain: 50.000000 dB...<br />
Actual RX Gain: 50.000000 dB...<br />
<br />
Checking RX: all_los: locked ...<br />
<br />
Done!<br />
</pre><br />
<br />
===Benchmarking your system===<br />
Included with the UHD driver example programs is a utility, <code>benchmark_rate</code> to benchmark the transport link of the system. <br />
<br />
A system's maximum performance is dependent upon many factors. <code>benchmark_rate</code> will exercise the transport link and CPU of the system. <br />
<br />
====1 Gb Interface====<br />
NOTE: This example requires the <code>HG</code> FPGA image to be loaded.<br />
<br />
'''N300/N310'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 3.84 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=122.88e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 3.84e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 3.84e6 \<br />
--tx_subdev "A:0"<br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 1.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 1.25e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 1.25e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
'''N320/N321'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 3.84 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=245.76e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 3.84e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 3.84e6 \<br />
--tx_subdev "A:0"<br />
<br />
When streaming samples over a 1 Gb transport link, the maximum accumulative rate for all channels is 25 MS/s with a <code>sc16</code> OTW format. To achieve higher streaming rates, it is recommended to use the 10 Gb interfaces.<br />
<br />
====10 Gb Interface SFP 1====<br />
NOTE: This example will work with either the <code>HG</code> or <code>XG</code> FPGA image.<br />
<br />
'''N300/N310'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 31.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 31.25e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 31.25e6 \<br />
--tx_subdev "A:0" <br />
<br />
'''N320/N321'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 31.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=250e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 31.25e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 31.25e6 \<br />
--tx_subdev "A:0" <br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 30.72 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=122.88e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 30.72e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 30.72e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
'''N320/N321'''<br />
<br />
This example will test two full-duplex streams at 30.72 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=245.76e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 30.72e6 \<br />
--rx_subdev "A:0 B:0" \<br />
--tx_rate 30.72e6 \<br />
--tx_subdev "A:0 B:0"<br />
<br />
====Dual 10 Gb Interface====<br />
NOTE: This example requires the <code>XG</code> FPGA image to be loaded.<br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 62.5 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 62.5e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 62.5e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
<br />
'''N320/N321'''<br />
<br />
This example will test two full-duplex streams at 62.5 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=250e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 62.5e6 \<br />
--rx_subdev "A:0 B:0" \<br />
--tx_rate 62.5e6 \<br />
--tx_subdev "A:0 B:0"<br />
<br />
==USRP N3xx Device Specific Operations==<br />
<br />
===White Rabbit Ethernet-Based Synchronization===<br />
* [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]]<br />
<br />
<br />
===N320/N321===<br />
* [[USRP N320/N321 LO Distribution]]<br />
* [[5G NR EVM Measurements with the USRP N320/N321]]<br />
<br />
<br />
===Turning the Device Off/On===<br />
To avoid damaging the file system and causing any corruption, do not turn the device off with the power button without first shutting down the system. Use this command to cleanly and properly shut the system down:<br />
<br />
shutdown -h now<br />
<br />
===Enable Auto Booting===<br />
Auto booting of the N3xx when power is applied can be configured by enabling the flag on the device's EEPROM with the following command:<br />
<br />
eeprom-set-flags 0x1<br />
<br />
===Default Password===<br />
The default user is <code>root</code> and the password is empty (no password).<br />
<br />
It is recommended to update the <code>root</code> password, which can be done with the command <code>passwd</code>:<br />
<br />
Example Output:<br />
<br />
root@ni-n3xx-serial:~# passwd<br />
Changing password for root<br />
New password: <br />
Re-enter new password: <br />
passwd: password changed.<br />
<br />
==Technical Support and Community Knowledge Base==<br />
Technical support for USRP hardware is available through email only. If the product arrived in a nonfunctional state or you require technical assistance, please contact [mailto:support@ettus.com support@ettus.com]. Please allow 24 to 48 hours for response by email, depending on holidays and weekends, although we are often able to reply more quickly than that.<br />
<br />
We also recommend that you subscribe to the community mailing lists. The mailing lists have a responsive and knowledgeable community of hundreds of developers and technical users who are located around the world. When you join the community, you will be connected to this group of people who can help you learn about SDR and respond to your technical and specific questions. Often your question can be answered quickly on the mailing lists. Each mailing list also provides an archive of all past conversations and discussions going back many years. Your question or problem may have already been addressed before, and a relevant or helpful solution may already exist in the archive.<br />
<br />
Discussions involving the USRP hardware and the UHD software itself are best addressed through the '''usrp-users''' mailing list at [http://usrp-users.ettus.com http://usrp-users.ettus.com].<br />
<br />
Discussions involving the use of [http://gnuradio.org/ GNU Radio] with USRP hardware and UHD software are best addressed through the '''discuss-gnuradio''' mailing list at [https://lists.gnu.org/mailman/listinfo/discussgnuradio https://lists.gnu.org/mailman/listinfo/discussgnuradio].<br />
<br />
Discussions involving the use of [http://openbts.org/ OpenBTS®] with USRP hardware and UHD software are best addressed through the '''openbts-discuss''' mailing list at [https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss].<br />
<br />
The support page on our website is located at [https://www.ettus.com/support https://www.ettus.com/support]. The Knowledge Base is located at [https://kb.ettus.com https://kb.ettus.com].<br />
<br />
==Legal Considerations==<br />
Every country has laws governing the transmission and reception of radio signals. Users are solely responsible for insuring they use their USRP system in compliance with all applicable laws and regulations. Before attempting to transmit and/or receive on any frequency, we recommend that you determine what licenses may be required and what restrictions may apply.<br />
<br />
*NOTE: This USRP product is a piece of test equipment.<br />
<br />
==Sales and Ordering Support==<br />
If you have any non-technical questions related to your order, then please contact us by email at [mailto:orders@ettus.com orders@ettus.com], or by phone at +14086106399 (Monday-Friday, 8 AM - 5 PM, Pacific Time). Please be sure to include your order number and the serial number of your USRP.<br />
<br />
==Terms and Conditions of Sale==<br />
Terms and conditions of sale can be accessed online at the following link: http://www.ettus.com/legal/terms-and-conditions-of-sale<br />
<br />
[[Category:Getting Started Guides]]<br />
[[Category:N300]]<br />
[[Category:N310]]<br />
<br />
[[Category:N320]]<br />
<br />
[[Category:N321]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=USRP_N300/N310/N320/N321_Getting_Started_Guide&diff=6023USRP N300/N310/N320/N321 Getting Started Guide2024-03-11T19:04:29Z<p>MichaelDickens: /* Mender Update Process */ differen mender commands for different UHD filesystem versions</p>
<hr />
<div>==Kit Contents==<br />
===N300===<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N300<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n300 kit.png|450px|center]]<br />
|}<br />
===N310===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N310<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n310 kit.png|500px|center]]<br />
|}<br />
<br />
===N320===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N320<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
|[[File:n320 kit.png|500px|center]] <br />
|}<br />
<br />
===N321===<br />
<br />
{|<br />
|style="vertical-align:top"|<br />
* USRP N321<br />
* DC Power Supply (12V, 7A)<br />
* 1 RJ45 – SFP+ Adapter<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to Micro USB-B Cable (1m)<br />
* Getting Started Guide<br />
* Ettus Research Sticker<br />
||[[File:n321 kit.png|500px|center]] <br />
|}<br />
==Verify the Contents of Your Kit==<br />
Ensure that your kit contains all the items listed above. If any items are missing, please contact sales@ettus.com immediately.<br />
<br />
==You Will Need==<br />
* microSD Card Writer<br />
<br />
* For Network Mode: A host computer with an available 1 or 10 Gigabit Ethernet interface for sample streaming. In addition to the Ethernet interface used for sampling streaming, your host computer will require a separate 1 Gigabit Ethernet interface for command and control streaming.<br />
<br />
* For Stand-Alone Embedded Mode: A host computer with an available 1 Gigabit Ethernet port or a USB 2.0 port to remotely access the embedded Linux operating system running on ARM CPU.<br />
<br />
==Proper Care and Handling==<br />
All Ettus Research products are individually tested before shipment. The USRP is guaranteed to be functional at the time it is received by the customer. Improper use or handling of the USRP can cause the device to become non-functional. Take the following precautions to prevent damage to the unit.<br />
<br />
* Never allow metal objects to touch the circuit board while powered.<br />
* Always properly terminate the transmit port with an antenna or 50Ω load.<br />
* Always handle the board with proper anti-static methods.<br />
* Never allow the board to directly or indirectly come into contact with any voltage spikes.<br />
* Never allow any water or condensing moisture to come into contact with the device.<br />
* Always use caution with FPGA, firmware, or software modifications.<br />
<br />
{|<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |Never apply more than -15 dBm of power into any RF input.<br />
|-<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |Always use at least 30dB attenuation if operating in loopback configuration<br />
|-<br />
|}<br />
<br />
==Install and Setup the Software Tools on Your Host Computer==<br />
In order to use your Universal Software Radio Peripheral (USRP™), you must have the software tools correctly installed and configured on your host computer. A step-by-step guide for doing this is available at the Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux|Linux]], [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X|OS X]] and [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows|Windows]] Application Notes.<br />
<br />
To find the latest release of UHD, see the UHD repository at https://github.com/EttusResearch/uhd.<br />
<br />
The USRP N310 requires UHD version 3.11.0.0 or later. <br />
<br />
The USRP N300 requires UHD version 3.12.0.0 or later.<br />
<br />
The USRP N320/N321 requires UHD version 3.14.0.0 or later. <br />
<br />
White Rabbit Ethernet-Based Synchronization of the N3xx USRP requires UHD version 3.12.0.0 or later. For additional details on White Rabbit Ethernet-Based Synchronization, please see the application note, [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]].<br />
<br />
'''When you receive a brand-new device, it is strongly recommended that you download the most recent filesystem image from the Ettus Research website and write it to the SD card that comes with the unit. It is not recommended that you use the SD card from the factory as-is. Instructions on downloading the latest filesystem image and writing it to the SD card are listed below.'''<br />
<br />
'''Note that if you are operating the device in Network Mode, the version of UHD running on the host computer and the USRP N3xx must match.'''<br />
<br />
==Connecting the Device==<br />
===Interfaces Overview===<br />
Listed below are the interfaces to connect to the USRP N3xx. Each interface has specific functionality, limitations and purpose. <br />
<br />
'''Serial Console'''<br />
<br />
The Serial Console provides a low level interface to the device typically used for debugging.<br />
<br />
'''1 Gigabit RJ45 Connection'''<br />
<br />
The 1 Gigabit RJ45 Connection interfaces with the on-board ARM CPU. When operated in "Network mode", this interface can optionally be used for UHD management traffic. Regardless of the operation mode (Network vs Embedded) this interface can be used to connect to the ARM via SSH. By default, the 1Gb RJ45 connection is configured to use a DHCP assigned IP address.<br />
<br />
'''Dual SFP+ Connections'''<br />
<br />
The Dual SFP+ Connections support multiple configurations for streaming high-speed, low-latency data, depending upon the FPGA image which is loaded.<br />
<br />
'''QSFP+ Connection (N320/ N321 Only)'''<br />
<br />
The QSFP+ Connection supports 2 x 10Gb lanes for streaming high-speed, low-latency data, while the onboard SFP0 connection is used for White Rabbit Ethernet-Based Synchronization.<br />
<br />
===Setting up a Serial Console Connection===<br />
It is possible to gain shell access to the device using a serial terminal emulator via the Serial Console port. Most Linux, OSX, or other Unix based operating systems have a tool called <code>screen</code> which can be used for this purpose. <br />
<br />
If you do not have <code>screen</code> installed, it can be installed via your package manager. For Ubuntu/Debian based operating systems it can be installed with <code>apt</code> such as:<br />
<br />
sudo apt install screen<br />
<br />
The default Baud Rate for the Serial Console is: <code>115200</code><br />
<br />
The exact device node you should attach to depends on your operating system's driver and other USB devices that might already be connected. Modern Linux systems offer alternatives to simply trying device nodes; instead, the OS might have a directory of symlinks under <code>/dev/serial/by-id</code>:<br />
<br />
$ ls /dev/serial/by-id<br />
usb-Digilent_Digilent_USB_Device_25163511FE00-if00-port0<br />
usb-Digilent_Digilent_USB_Device_25163511FE00-if01-port0<br />
usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if00-port0<br />
usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if01-port0<br />
<br />
NOTE: Exact names depend on the host operating system version and may differ.<br />
<br />
Every N3XX series device connected to USB will by default show up as four different devices. The devices labeled <code>"USB_to_UART_Bridge_Controller"</code> are the devices that offer a serial prompt. The first (with the <code>if00</code> suffix) connects to the <code>ARM CPU</code>, whereas the second connects to the <code>STM32 Microcontroller</code>. <br />
<br />
If you have multiple N3xx Serial Consoles connected to a single host, you may have to empirically test nodes. <br />
<br />
Connecting to the ARM CPU can be performed with the command:<br />
<br />
$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if00-port0 115200<br />
<br />
Upon starting the USRP N3xx, boot messages will appear and rapidly update. Once the boot process successfully completes, a login prompt like the following should appear:<br />
<br />
OpenEmbedded test ni-n3xx-313ABDA ttyPS0<br />
<br />
ni-n3xx-313ABDA login: <br />
<br />
Enter the username: <code>root</code> <br />
<br />
By default, the <code>root</code> user's password is left blank. Press the <code>Enter</code> key when prompted for a password.<br />
<br />
You should now be presented with a shell prompt similar to the following:<br />
<br />
root@ni-n3xx-<motherboard serial #>:~#<br />
<br />
Using the default configuration, the serial console will show all kernel log messages (which are not available when using SSH), and give access to the boot loader (U-boot prompt). This can be used to debug kernel or boot-loader issues more efficiently than when logged in via SSH.<br />
<br />
====Connecting to the microcontroller====<br />
<br />
Using the Serial Console interface, it is possible to connect to the STM32 microcontroller with the command below. The STM32 controls the power sequencing and several other low level device operations.<br />
<br />
$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_007F6CB5-if01-port0 115200<br />
<br />
The STM32 interface provides a very simple prompt. The command <code>help</code> will list all available commands. A direct connection to the microcontroller can be used to hard-reset the device without physically accessing it (i.e., emulating a power button press) and other low-level diagnostics.<br />
<br />
===Connecting to the ARM via SSH===<br />
By default, the RJ45 1Gb management interface is configured to be assigned a DHCP IP address. <br />
<br />
If you have access to a network which provides a DHCP server (such as a common router's LAN), attach the RJ45 1Gb port to this network. Details vary by vendor, however, most router management interfaces will provide a list of attached devices to the LAN including their IP address.<br />
<br />
Without access to a router management interface, you can identify the IP address by connecting to the ARM CPU via Serial Console as detailed in the section above and running the command <code>ip a</code>:<br />
<br />
Example Output:<br />
<br />
<pre><br />
# ip a<br />
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000<br />
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br />
inet 127.0.0.1/8 scope host lo<br />
valid_lft forever preferred_lft forever<br />
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
inet 192.168.1.151/24 brd 192.168.1.255 scope global dynamic eth0<br />
valid_lft 42865sec preferred_lft 42865sec<br />
3: sfp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
inet 192.168.10.2/24 brd 192.168.10.255 scope global sfp0<br />
valid_lft forever preferred_lft forever<br />
4: sfp1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc pfifo_fast qlen 1000<br />
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff<br />
</pre><br />
<br />
If you do not have access to a network with a DHCP server, you can create one using the Linux utility <code>dnsmasq</code>:<br />
<br />
$ sudo dnsmasq -i <ETHERNET_ADAPTER_NAME> --dhcp-range=192.168.1.151,192.168.1.254 --except-interface=lo --bind-dynamic --no-daemon<br />
<br />
NOTE: Modify the value <code><ETHERNET_ADAPTER_NAME></code> to match the interface you would like to create a DHCP server on.<br />
<br />
After the device has obtained an IP address, you can remotely log into it from a Linux or macOS system with SSH, as shown below:<br />
<br />
$ ssh root@192.168.1.151<br />
<br />
NOTE: The IP address may vary depending on your network setup.<br />
<br />
NOTE: The <code>root</code> password default password is empty/blank.<br />
<br />
On Microsoft Windows, the SSH connection can be established using the third-party program Putty. <br />
<br />
After logging in, you should be presented with a shell like the following:<br />
<br />
root@ni-n3xx-<motherboard serial #>:~#<br />
<br />
==Updating the Linux File System==<br />
Before operating the device, it is strongly recommended to update to the latest version of the Embedded Linux file system. If you are operating the device in Network Mode, the version of UHD running on the host machine and N3xx USRP must match. <br />
<br />
There is two ways to update the file system for the N3xx USRP: <br />
<br />
1. Mender<br />
<br />
2. Physically remove microSD card from device and write a new file system to the microSD card. <br />
<br />
===File System Partition Layout===<br />
The SD Card is divided into four partitions. There is two root file system partitions, a boot partition and a data partition. <br />
<br />
Any data you would like to preserve through Mender updates should be saved to the <code>data</code> partition, which is mounted at <code>/data</code>.<br />
<br />
===Updating the file system with Mender===<br />
Mender is third-party software that enables remote updating of the root file system without physically accessing the device (see also the Mender website https://mender.io). Mender can be executed locally on the device, or a Mender server can be set up which can be used to remotely update an arbitrary number of USRP devices. Users can host their own local Mender server, or use servers hosted by Mender as a paid service; contact Mender for more information. <br />
<br />
====Mender Update Process====<br />
<br />
When updating the file system using Mender, the tool will overwrite the root file system partition that is not currently mounted. Any data stored in the root partitions will be permanently lost with a Mender update.<br />
<br />
After updating a partition with Mender, it will reboot into the newly updated partition. Only if the update is confirmed by the user, the update will be made permanent. This means that if an update fails, the device will be always able to reboot into the partition from which the update was originally launched, which presumably is in a working state. Another update can be launched now to correct the previous, failed update, until it works.<br />
<br />
To obtain the file system Mender image (these are files with a <code>.mender</code> suffix), run the following command on the host computer with Internet access:<br />
$ sudo uhd_images_downloader -t mender -t n3xx --yes<br />
<br />
Example Output:<br />
[INFO] Using base URL: https://files.ettus.com/binaries/cache/<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
365684 kB / 365684 kB (100%) n3xx_common_mender_default-v4.6.0.0.zip<br />
[INFO] Images download complete.<br />
<br />
The downloaded "zip" archive is extracted into the <code>Images destination</code> directory with the filename <code>usrp_n3xx_fs.mender</code>. Next, you will need to copy this Mender file system image from the <code>Images destination</code> directory to the USRP N3xx to have its filesystem changed. This can be done with the Linux utility <code>scp</code>, for example as follows:<br />
<br />
$ scp /usr/local/share/uhd/images/usrp_n3xx_fs.mender root@192.168.1.51:~/. <br />
<br />
Note: The path and IP may be different for your configuration; the command above assumes you're using the default UHD installation path of <code>/usr/local</code> and that the N3xx's IP is <code>192.168.1.51</code>.<br />
<br />
After copying the Mender file system image to the N3xx, connect to the N3xx to gain shell access via either the Serial Console or SSH.<br />
<br />
On the N3xx, you first need to determine the version of UHD currently running on the USRP; an easy way to do this is via the command<br />
$ uhd_config_info --version<br />
<br />
Example output:<br />
UHD 3.14.1.1-0-g0347a6d8<br />
<br />
The mender command to execute is different for UHD version 4.0 or newer versus prior to version 4.0. For the former use <code>mender install</code> followed by the mender file; for the latter use <code>mender -f -rootfs</code> followed by the mender file. Starting with UHD version 4.0 one can use mender to upgrade or downgrade the UHD filesystem version between any UHD v4 versions (e.g., 4.1 to 4.6; 4.6 to 4.1). The following commands assume that the UHD filesystem is version 4; if not then substitute the other mender command.<br />
<br />
Run <code>mender install /path/to/latest.mender</code> to update the file system, e.g.:<br />
<br />
$ mender install usrp_n3xx_fs.mender<br />
<br />
The artifact can also be stored on a remote server:<br />
$ mender install <nowiki>http://server.name/path/to/latest.mender</nowiki><br />
<br />
This procedure will take a few minutes to complete. After mender has logged a successful update, reboot the device:<br />
$ reboot<br />
<br />
If the reboot worked, and the device seems functional, commit the changes so that the boot loader knows to permanently boot into this partition:<br />
$ mender commit<br />
<br />
To identify the currently installed Mender artifact from the command line, the following file can be queried on the N3xx:<br />
$ cat /etc/mender/artifact_info<br />
<br />
If you are using a Mender server, the updates can be initiated from a web dashboard. From there, you can start the updates without having to log into the device, and you can update groups of USRPs with a few clicks in a web GUI. The dashboard can also be used to inspect the state of USRPs. This is a simple way to update groups of rack-mounted USRPs with custom file systems.<br />
<br />
For more information on updating the filesystem, refer to the [https://files.ettus.com/manual/ UHD Manual].<br />
<br />
===Updating the files system by writing the disk image===<br />
Please see the separate application note, [[Writing the USRP File System Disk Image to a SD Card]], for step-by-step instructions on writing the file system image to the SD card.<br />
<br />
==Updating the Network Configurations==<br />
The USRP N3xx systemd network configuration files are located at: <code>/data/network/</code><br />
<br />
# ls /data/network/<br />
eth0.network int0.network sfp0.network sfp1.network<br />
<br />
or for older versions of the file system: <code>/etc/systemd/network/</code><br />
<br />
# ls /etc/systemd/network/<br />
eth0.network sfp0.network sfp1.network<br />
<br />
For details on configuration please refer to the [https://www.freedesktop.org/software/systemd/man/systemd.network.html systemd-networkd manual pages].<br />
<br />
The factory settings are as follows:<br />
<pre><br />
eth0 (DHCP):<br />
<br />
[Match]<br />
Name=eth0<br />
<br />
[Network]<br />
DHCP=ipv4<br />
<br />
[DHCP]<br />
UseHostname=false<br />
<br />
sfp0 (static):<br />
<br />
[Match]<br />
Name=sfp0<br />
<br />
[Network]<br />
Address=192.168.10.2/24<br />
<br />
[Link]<br />
MTUBytes=9000<br />
<br />
sfp1 (static):<br />
<br />
[Match]<br />
Name=sfp1<br />
<br />
[Network]<br />
Address=192.168.20.2/24<br />
<br />
[Link]<br />
MTUBytes=9000<br />
</pre><br />
<br />
Additional notes on networking:<br />
<br />
* Care needs to be taken when editing these files on the device, since <code>vi</code> / <code>vim</code> sometimes generates undo files (e.g. <code>/etc/systemd/network/sfp0.network~</code>), that <code>systemd-networkd</code> might accidentally pick up.<br />
* Temporarily setting the IP addresses or MTU sizes via <code>ifconfig</code> or other command line tools will only change the value until the next reboot or reload of the FPGA image.<br />
* If the MTU of the device and host computers differ, streaming issues can occur.<br />
* Streaming via SFP0 at 1 Gb rates requires a MTU of <code>1500</code><br />
* Streaming via SFP0 at 10 Gb rates requires a MTU of <code>9000</code><br />
<br />
For addition details on network configuration here: https://files.ettus.com/manual/page_usrp_n3xx.html#n3xx_network_configuration<br />
<br />
==Updating the FPGA Image==<br />
<br />
===Network Mode FPGA Image Update===<br />
The FPGA image should match the version of UHD installed on the host computer, when operated in Network mode. Connect the device to the host computer using either the RJ45 or SFP+ port, refer to the section above for detailed instructions. <br />
<br />
To obtain all the FPGA images for a specific version of UHD, run the following command on the host computer with internet access:<br />
<br />
$ sudo uhd_images_downloader<br />
<br />
Example Output:<br />
<br />
$ sudo uhd_images_downloader<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
00006 kB / 00006 kB (100%) usrp1_b100_fw_default-g6bea23d.zip<br />
19810 kB / 19810 kB (100%) x3xx_x310_fpga_default-gf1ba32fe.zip<br />
02757 kB / 02757 kB (100%) usrp2_n210_fpga_default-g6bea23d.zip<br />
02123 kB / 02123 kB (100%) n230_n230_fpga_default-ge57dfe0.zip<br />
00522 kB / 00522 kB (100%) usrp1_b100_fpga_default-g6bea23d.zip<br />
00491 kB / 00491 kB (100%) b2xx_b200_fpga_default-ge57dfe0.zip<br />
02415 kB / 02415 kB (100%) usrp2_n200_fpga_default-g6bea23d.zip<br />
08988 kB / 08988 kB (100%) e3xx_e320_fpga_default-g3de8954a.zip<br />
23045 kB / 23045 kB (100%) n3xx_n310_fpga_default-g3de8954a.zip<br />
00523 kB / 00523 kB (100%) b2xx_b205mini_fpga_default-ge57dfe0.zip<br />
18937 kB / 18937 kB (100%) x3xx_x300_fpga_default-gf1ba32fe.zip<br />
00017 kB / 00017 kB (100%) octoclock_octoclock_fw_default-g14000041.zip<br />
00007 kB / 00007 kB (100%) usrp2_usrp2_fw_default-g6bea23d.zip<br />
00009 kB / 00009 kB (100%) usrp2_n200_fw_default-g6bea23d.zip<br />
00450 kB / 00450 kB (100%) usrp2_usrp2_fpga_default-g6bea23d.zip<br />
00144 kB / 00144 kB (100%) b2xx_common_fw_default-ga69ab0c.zip<br />
25107 kB / 25107 kB (100%) n3xx_n320_fpga_default-g3de8954a.zip<br />
00464 kB / 00464 kB (100%) b2xx_b200mini_fpga_default-ge57dfe0.zip<br />
00319 kB / 00319 kB (100%) usrp1_usrp1_fpga_default-g6bea23d.zip<br />
04839 kB / 04839 kB (100%) usb_common_windrv_default-g14000041.zip<br />
00009 kB / 00009 kB (100%) usrp2_n210_fw_default-g6bea23d.zip<br />
16065 kB / 16065 kB (100%) n3xx_n300_fpga_default-g3de8954a.zip<br />
05578 kB / 05578 kB (100%) e3xx_e310_fpga_default-g4bc2c6f.zip<br />
00885 kB / 00885 kB (100%) b2xx_b210_fpga_default-ge57dfe0.zip<br />
[INFO] Images download complete.<br />
<br />
<br />
NOTE: In the above example output, the Images Destination folder is printed:<br />
<br />
[INFO] Images destination: /usr/local/share/uhd/images<br />
<br />
To list the N3xx FPGA images with a full path, run the command:<br />
<br />
$ ls -w 1 /usr/local/share/uhd/images/usrp_n3*.bit<br />
<br />
/usr/local/share/uhd/images/usrp_n300_fpga_AA.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n300_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_AA.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n310_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_AQ.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_HG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_WX.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_XG.bit<br />
/usr/local/share/uhd/images/usrp_n320_fpga_XQ.bit<br />
<br />
To update the default <code>HG</code> variant of FPGA image, run the command:<br />
<br />
$ uhd_image_loader --args "type=n3xx,addr=<N3xx_IP_ADDR>,fpga=HG"<br />
<br />
Example Output:<br />
<br />
uhd_image_loader --args "type=n3xx,addr=192.168.1.151,fpga=HG"<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.11.1.HEAD-0-gad6b0935<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.151,type=n3xx,product=n310,serial=313ABDA,claimed=False,skip_init=1<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.11.1.0-gunknown<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager.UDP] No CHDR interfaces found!<br />
[INFO] [MPMD] Claimed device without full initialization.<br />
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.<br />
[INFO] [MPM.PeriphManager] Updating component `fpga'<br />
[INFO] [MPM.PeriphManager] Updating component `dts'<br />
[INFO] [MPM.RPCServer] Resetting peripheral manager.<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.<br />
<br />
<br />
To load a different default FPGA image (i.e. <code>XG</code>, <code>WG</code>), modify the device argument <code>fpga=</code> to a value of <code>fpga=XG</code> or <code>fpga=WG</code>.<br />
<br />
To specify the path to a custom FPGA image, use the <code>--fpga-path</code> argument. <br />
<br />
$ uhd_image_loader --args "type=n3xx,addr=<N3xx_IP_ADDR>" --fpga-path=/path/to/custom/fpga.bit<br />
<br />
The Verilog code for the FPGA in the USRP N3xx is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
<br />
===Embedded Mode FPGA Image Update===<br />
<br />
It is possible to update the FPGA image when operated in Embedded mode. Connect to the ARM CPU via Serial Console or SSH as detailed in the section above. <br />
<br />
Updating the FPGA image from the ARM CPU is the same as detailed above for a Network mode update, except it is not required to provide an <code>addr</code> device argument. <br />
<br />
uhd_image_loader --args "type=n3xx,fpga=HG"<br />
<br />
<pre><br />
root@ni-n3xx-313ABDA:~# uhd_image_loader --args "type=n3xx,fpga=HG"<br />
[INFO] [UHD] linux; GNU C++ version 7.2.0; Boost_106400; UHD_3.11.1.0-0-unknown<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=127.0.0.1,type=n3xx,product=n310,serial=313ABDA,claimed=False,skip_init=1<br />
[INFO] [MPMD] Claimed device without full initialization.<br />
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.<br />
[INFO] [MPM.PeriphManager] Updating component `fpga'<br />
[INFO] [MPM.PeriphManager] Updating component `dts'<br />
[INFO] [MPM.RPCServer] Resetting peripheral manager.<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.<br />
</pre><br />
<br />
For more information on updating the FPGA image, refer to the UHD Manual at http://uhd.ettus.com .<br />
<br />
==Setting Up a Streaming Connection==<br />
The device supports multiple, high-speed, low-latency interfaces on the SFP+ ports for streaming samples to the host computer. <br />
<br />
===1Gb Streaming SFP Port 0===<br />
Complete the steps below to set up a streaming connection over the 1 Gigabit Ethernet interface on <code>SFP Port 0</code>.<br />
<br />
When streaming via SFP Port 0 at 1 Gb speeds, it is important that the connection is direct between the Host and USRP. Placing a switch or other network gear between the Host and USRP can reduce throughput of the transport link. It is also generally recommended to avoid using USB to Ethernet Adapters for the high speed streaming interface, as they may limit performance or cause periodic flow control errors. <br />
<br />
NOTE: The <code>HG</code> FPGA image must be loaded for <code>SFP Port 0</code> to operate at 1Gb speeds. If the <code>XG</code> image is loaded, the port will be unresponsive at 1Gb speeds. <br />
<br />
1. Configure your Host's Ethernet adapter as shown below. This interface should be separate from the 1Gb NIC/network which is connected to the 1Gb RJ45 management interface.<br />
<br />
IP Address: 192.168.10.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 1500<br />
<br />
NOTE: When operating <code>SFP Port 0</code> at 1Gb speeds, it is important to set a MTU of <code>1500</code> and not a value of <code>automatic</code>.<br />
<br />
2. Insert the RJ45 – SFP+ adapter into <code>SFP Port 0</code> .<br />
<br />
3. Connect the adapter to a host computer using the Ethernet cable to SFP0.<br />
<br />
The Green LED above <code>SFP Port 0</code> should illuminate.<br />
<br />
4. To test the connection, <code>ping</code> the device at address <code>192.168.10.2</code> from the host, as shown<br />
below:<br />
<br />
$ ping 192.168.10.2<br />
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.<br />
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=1.06 ms<br />
^C<br />
--- 192.168.10.2 ping statistics ---<br />
1 packets transmitted, 1 received, 0% packet loss, time 0ms<br />
rtt min/avg/max/mdev = 1.065/1.065/1.065/0.000 ms<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
===10Gb Streaming SFP Port 1===<br />
Complete the steps below to set up a streaming connection over the 10 Gigabit Ethernet interface on <code>SFP Port 1</code>.<br />
<br />
NOTE: Both the <code>HG</code> and <code>XG</code> FPGA images support 10Gb speeds over SFP Port 1. <br />
<br />
1. Configure your Host's 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.20.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
NOTE: When operating at 10Gb speeds, it is important to set a MTU of <code>9000</code> and not a value of <code>automatic</code>.<br />
<br />
2. Connect the USRP to a host computer using either a 10Gb SFP or Fiber cable to <code>SFP Port 1</code>.<br />
<br />
The Green LED above <code>SFP Port 1</code> should illuminate.<br />
<br />
3. To test the connection, <code>ping</code> the device at address <code>192.168.20.2</code> from the host, as shown<br />
below:<br />
<br />
$ ping 192.168.20.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
===Dual 10Gb Streaming SFP Ports 0/1===<br />
Complete the steps below to set up a streaming connections over the Dual 10 Gigabit Ethernet interface on <code>SFP Ports 0/1</code>.<br />
<br />
NOTE: The <code>XG</code> FPGA image must be loaded for <code>SFP Port 0</code> to operate at 10 Gb speeds. If the <code>HG</code> image is loaded, the port will be unresponsive at 10 Gb speeds. <br />
<br />
1. Configure your Host's #1 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.10.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
2. Configure your Host's #2 10Gb Ethernet adapter as shown below. <br />
<br />
IP Address: 192.168.20.1<br />
Subnet Mask: 255.255.255.0<br />
Gateway: 0.0.0.0<br />
MTU: 9000<br />
<br />
NOTE: When operating at 10Gb speeds, it is important to set a MTU of <code>9000</code> and not a value of <code>automatic</code>.<br />
<br />
3. Connect the USRP to a host computer using either a 10Gb SFP or Fiber cables to <code>SFP Ports 0/1</code>.<br />
<br />
The Green LEDs above <code>SFP Ports 0/1</code> should illuminate.<br />
<br />
4. To test the <code>SFP Port 0</code> connection, <code>ping</code> the device at address <code>192.168.10.2</code> from the host, as shown below:<br />
<br />
$ ping 192.168.10.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
5. To test the <code>SFP Port 1</code> connection, <code>ping</code> the device at address <code>192.168.20.2</code> from the host, as shown below:<br />
<br />
$ ping 192.168.20.2<br />
<br />
Press <code>CTRL+C</code> to stop the ping program. <br />
<br />
Proceed to the next section "Verifying Device Operation".<br />
<br />
For more details on Network Setup and Configuration, please see the “Interfaces and Connectivity” section on the [[N300/N310]] or [[N320/N321]] hardware resources pages.<br />
<br />
==Verifying Device Operation==<br />
Once you have successfully setup a management interface and streaming interface, you can now verify the devices operation using the included UHD utilities.<br />
<br />
===Subdevice Specification Mapping===<br />
====N300====<br />
The USRP N300 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = A:1<br />
<br />
====N310====<br />
The USRP N310 contains 4 channels, each represented on the front panel as <code>RF0-3</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
=====UHD 3.11.x.x - 3.12.x.x=====<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
* RF2 = C:0<br />
* RF3 = D:0<br />
<br />
=====UHD 3.13.x.x+=====<br />
* RF0 = A:0<br />
* RF1 = A:1<br />
* RF2 = B:0<br />
* RF3 = B:1<br />
<br />
====N320====<br />
The USRP N320 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
<br />
====N321====<br />
The USRP N321 contains 2 channels, each represented on the front panel as <code>RF0-1</code>. Below is the <code>subdev</code> mapping of RF Ports.<br />
<br />
* RF0 = A:0<br />
* RF1 = B:0<br />
<br />
Additional details of UHD Subdevice Specifications can be found here in the UHD Manual: http://files.ettus.com/manual/page_configuration.html#config_subdev<br />
<br />
===Supported Sample Rates===<br />
<br />
The USRP N300/N310 supports the three fixed Master Clock Rates listed below. <br />
<br />
* 122.88 MHz<br />
* 125.00 MHz<br />
* 153.60 MHz<br />
<br />
The USRP N320/N321 supports the three fixed Master Clock Rates listed below. <br />
<br />
* 200.00 MHz<br />
* 245.76 MHz<br />
* 250.00 MHz<br />
<br />
Sample rates as delivered to/from the host computer for USRP devices are constrained to follow several important rules.<br />
<br />
It is important to understand that strictly-integer decimation and interpolation are used within USRP hardware to meet the requested sample rate requirements of the application at hand. That means that the desired sample rate must meet the requirement that master-clock-rate/desired-sample-rate be an integer ratio. Further, it is strongly desirable for that ratio to be even. This ratio is the decimation (down-conversion) or interpolation (up-conversion) factor. The decimation or interpolation factor may be between 1 and 1024. There are further constraints on the decimation or interpolation factor. If the decimation or interpolation factor exceeds 128, then it must be evenly divisible by 2. If the decimation or interpolation factor exceeds 256, then it must be evenly divisible by 4.<br />
<br />
====Example Sample Rates====<br />
Listed below are common sample rates for the given master clock rates. This is not a complete listing of the supported sample rates.<br />
<br />
{| class="wikitable"<br />
!Master Clock Rate<br />
!colspan="20"|Decimation / Interpolation Rate <br> Host Sample Rate [Msps]<br />
|-<br />
<br />
|style="text-align:center;"| 1<br />
|style="text-align:center;"| 2<br />
|style="text-align:center;"| 4<br />
|style="text-align:center;"| 6<br />
|style="text-align:center;"| 8<br />
|style="text-align:center;"| 10<br />
|style="text-align:center;"| 12<br />
|style="text-align:center;"| 14<br />
|style="text-align:center;"| 16<br />
|style="text-align:center;"| 18<br />
|style="text-align:center;"| 20<br />
|style="text-align:center;"| 30<br />
|style="text-align:center;"| 32<br />
|style="text-align:center;"| 64<br />
|style="text-align:center;"| 100<br />
|style="text-align:center;"| 128<br />
|style="text-align:center;"| 200<br />
|style="text-align:center;"| 256<br />
|style="text-align:center;"| 512<br />
|style="text-align:center;"| 1024<br />
|-<br />
<br />
|style="text-align:center;"| 122.88e6<br />
|style="text-align:center;"| 61.44e6<br />
|style="text-align:center;"| 30.72e6<br />
|style="text-align:center;"| 20.48e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.288e6<br />
|style="text-align:center;"| 10.24e6<br />
|style="text-align:center;"| 8.7771e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 6.8267e6<br />
|style="text-align:center;"| 6.144e6<br />
|style="text-align:center;"| 4.096e6<br />
|style="text-align:center;"| 3.84e6<br />
|style="text-align:center;"| 1.92e6<br />
|style="text-align:center;"| 1.2288e6<br />
|style="text-align:center;"| 960e3<br />
|style="text-align:center;"| 614.4e3<br />
|style="text-align:center;"| 480e3<br />
|style="text-align:center;"| 240e3<br />
|style="text-align:center;"| 120e3<br />
|-<br />
<br />
|style="text-align:center;"| 125e6<br />
|style="text-align:center;"| 62.5e6<br />
|style="text-align:center;"| 31.25e6<br />
|style="text-align:center;"| 20.833e6<br />
|style="text-align:center;"| 15.625e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 10.417e6<br />
|style="text-align:center;"| 8.9286e6<br />
|style="text-align:center;"| 7.8125e6<br />
|style="text-align:center;"| 6.9444e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 4.1667e6<br />
|style="text-align:center;"| 3.90625e6<br />
|style="text-align:center;"| 1.953125e6<br />
|style="text-align:center;"| 1.25e6<br />
|style="text-align:center;"| 976.5625e3<br />
|style="text-align:center;"| 625e3<br />
|style="text-align:center;"| 488.28125e3<br />
|style="text-align:center;"| 244.14e3<br />
|style="text-align:center;"| 122.07e3<br />
<br />
|-<br />
<br />
|style="text-align:center;"| 153.6e6<br />
|style="text-align:center;"| 76.8e6<br />
|style="text-align:center;"| 38.4e6<br />
|style="text-align:center;"| 25.6e6<br />
|style="text-align:center;"| 19.2e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.8e6<br />
|style="text-align:center;"| 10.971e6<br />
|style="text-align:center;"| 9.6e6<br />
|style="text-align:center;"| 8.5333e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 5.12e6<br />
|style="text-align:center;"| 4.8e6<br />
|style="text-align:center;"| 2.4e6<br />
|style="text-align:center;"| 1.536e6<br />
|style="text-align:center;"| 1.2e6<br />
|style="text-align:center;"| 768e3<br />
|style="text-align:center;"| 600e3<br />
|style="text-align:center;"| 300e3<br />
|style="text-align:center;"| 150e3<br />
<br />
|-<br />
<br />
|}<br />
<br />
<br />
====N320/N321 Example Sample Rates====<br />
Listed below are common sample rates for the given master clock rates. This is not a complete listing of the supported sample rates.<br />
<br />
{| class="wikitable"<br />
!Master Clock Rate<br />
!colspan="20"|Decimation / Interpolation Rate <br> Host Sample Rate [Msps]<br />
|-<br />
<br />
|style="text-align:center;"| 1<br />
|style="text-align:center;"| 2<br />
|style="text-align:center;"| 4<br />
|style="text-align:center;"| 6<br />
|style="text-align:center;"| 8<br />
|style="text-align:center;"| 10<br />
|style="text-align:center;"| 12<br />
|style="text-align:center;"| 14<br />
|style="text-align:center;"| 16<br />
|style="text-align:center;"| 18<br />
|style="text-align:center;"| 20<br />
|style="text-align:center;"| 30<br />
|style="text-align:center;"| 32<br />
|style="text-align:center;"| 64<br />
|style="text-align:center;"| 100<br />
|style="text-align:center;"| 128<br />
|style="text-align:center;"| 200<br />
|style="text-align:center;"| 256<br />
|style="text-align:center;"| 512<br />
|style="text-align:center;"| 1024<br />
|-<br />
<br />
|style="text-align:center;"| 200e6<br />
|style="text-align:center;"| 100e6<br />
|style="text-align:center;"| 50e6<br />
|style="text-align:center;"| 33.33e6<br />
|style="text-align:center;"| 25e6<br />
|style="text-align:center;"| 20e6<br />
|style="text-align:center;"| 16.66e6<br />
|style="text-align:center;"| 14.2857e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 11.11e6<br />
|style="text-align:center;"| 10e6<br />
|style="text-align:center;"| 6.667e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 3.125e6<br />
|style="text-align:center;"| 2e6<br />
|style="text-align:center;"| 1.5625e6<br />
|style="text-align:center;"| 1e6<br />
|style="text-align:center;"| 781.25e3<br />
|style="text-align:center;"| 390.625e3<br />
|style="text-align:center;"| 195.3125e3<br />
<br />
|-<br />
<br />
|style="text-align:center;"| 245.76e6<br />
|style="text-align:center;"| 122.88e6<br />
|style="text-align:center;"| 61.44e6<br />
|style="text-align:center;"| 30.72e6<br />
|style="text-align:center;"| 20.48e6<br />
|style="text-align:center;"| 15.36e6<br />
|style="text-align:center;"| 12.288e6<br />
|style="text-align:center;"| 10.24e6<br />
|style="text-align:center;"| 8.7771e6<br />
|style="text-align:center;"| 7.68e6<br />
|style="text-align:center;"| 6.8267e6<br />
|style="text-align:center;"| 6.144e6<br />
|style="text-align:center;"| 4.096e6<br />
|style="text-align:center;"| 3.84e6<br />
|style="text-align:center;"| 1.92e6<br />
|style="text-align:center;"| 1.2288e6<br />
|style="text-align:center;"| 960e3<br />
|style="text-align:center;"| 614.4e3<br />
|style="text-align:center;"| 480e3<br />
|style="text-align:center;"| 240e3<br />
|-<br />
<br />
<br />
|style="text-align:center;"| 250e6<br />
|style="text-align:center;"| 125e6<br />
|style="text-align:center;"| 62.5e6<br />
|style="text-align:center;"| 31.25e6<br />
|style="text-align:center;"| 20.833e6<br />
|style="text-align:center;"| 15.625e6<br />
|style="text-align:center;"| 12.5e6<br />
|style="text-align:center;"| 10.417e6<br />
|style="text-align:center;"| 8.9286e6<br />
|style="text-align:center;"| 7.8125e6<br />
|style="text-align:center;"| 6.9444e6<br />
|style="text-align:center;"| 6.25e6<br />
|style="text-align:center;"| 4.1667e6<br />
|style="text-align:center;"| 3.90625e6<br />
|style="text-align:center;"| 1.953125e6<br />
|style="text-align:center;"| 1.25e6<br />
|style="text-align:center;"| 976.5625e3<br />
|style="text-align:center;"| 625e3<br />
|style="text-align:center;"| 488.28125e3<br />
|style="text-align:center;"| 244.14e3<br />
<br />
|-<br />
<br />
<br />
<br />
|}<br />
<br />
Additional information on Sample Rates can be found here in the UHD Manual: http://files.ettus.com/manual/page_general.html#general_sampleratenotes<br />
<br />
===Probe the USRP===<br />
<br />
====N300/N310====<br />
The UHD utility <code>uhd_usrp_probe</code> provides detailed information of the USRP device.<br />
<br />
From your host computer, run the command <code>uhd_usrp_probe</code>:<br />
<br />
<pre><br />
$ uhd_usrp_probe <br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.13.1.HEAD-0-ga0a71d10<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.10.2,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.13.1.0-gd3b7e90a<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal'.<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1355 MB/s)<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=192.168.10.2,clock_source=internal,time_source=internal,product=n310'.<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1358 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1355 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1345 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-313ABDA<br />
| | eeprom_version: 1<br />
| | mpm_version: 3.13.1.0-gd3b7e90a<br />
| | pid: 16962<br />
| | product: n310<br />
| | rev: 3<br />
| | rpc_connection: remote<br />
| | serial: 313ABDA<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.2<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_tpv, ref_locked, gps_time, gps_locked, temp, gps_sky, fan<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: AD9371 Dual ADC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX, RX2, CAL, LOCAL<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 75.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: AD9371 Dual ADC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: AD9371 Dual DAC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 1<br />
| | | | Name: Magnesium<br />
| | | | Antennas: TX/RX<br />
| | | | Sensors: lo_locked, ad9371_lo_locked, lowband_lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 65.0 step 0.5 dB<br />
| | | | Gain range rfic: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range dsa: 0.0 to 0.0 step 0.0 dB<br />
| | | | Gain range amp: 0.0 to 0.0 step 0.0 dB<br />
| | | | Bandwidth range: 20000000.0 to 100000000.0 step 0.0 Hz<br />
| | | | Connection Type: IQ<br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: AD9371 Dual DAC<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * DmaFIFO_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
<br />
<br />
<br />
</pre><br />
<br />
====N320====<br />
<br />
<pre><br />
<br />
$ uhd_usrp_probe <br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106600; UHD_3.14.0.0-0-g6875d061<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=127.0.0.1,type=n3xx,product=n320,serial=3181FFA,claimed=False<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.14.0.0-g6875d061<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 3181FFA<br />
[INFO] [MPM.Rhodium-0] Successfully loaded all peripherals!<br />
[INFO] [MPM.Rhodium-1] Successfully loaded all peripherals!<br />
[INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).<br />
[INFO] [MPM.PeriphManager] No QSFP board detected: Assuming it is disabled in the device tree overlay (e.g., HG, XG images).<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `time_source=internal,clock_source=internal'<br />
[INFO] [MPM.Rhodium-1] init() called with args `time_source=internal,clock_source=internal'<br />
[INFO] [MPM.Rhodium-0.init.LMK04828] LMK initialized and locked!<br />
[INFO] [MPM.Rhodium-1.init.LMK04828] LMK initialized and locked!<br />
[INFO] [MPM.Rhodium-1.DAC37J82] DAC PLL Locked!<br />
[INFO] [MPM.Rhodium-1.AD9695] ADC PLL Locked!<br />
[INFO] [MPM.Rhodium-1.init] JESD204B Link Initialization & Training Complete<br />
[INFO] [MPM.Rhodium-0.DAC37J82] DAC PLL Locked!<br />
[INFO] [MPM.Rhodium-0.AD9695] ADC PLL Locked!<br />
[INFO] [MPM.Rhodium-0.init] JESD204B Link Initialization & Training Complete<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'<br />
[INFO] [MPM.Rhodium-1] init() called with args `mgmt_addr=127.0.0.1,clock_source=internal,time_source=internal,product=n320'<br />
[INFO] [0/Replay_0] Initializing block control (NOC ID: 0x4E91A00000000004)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/FIFO_0] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
[INFO] [0/FIFO_1] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-3181FFA<br />
| | eeprom_version: 2<br />
| | mpm_version: 3.14.0.0-g6875d061<br />
| | pid: 16962<br />
| | product: n320<br />
| | rev: 6<br />
| | rpc_connection: local<br />
| | serial: 3181FFA<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.3<br />
| | FPGA git hash: 3de8954.clean<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_tpv, temp, gps_sky, fan, gps_time, gps_locked, ref_locked, gps_gpgga<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A79<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A67<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A79<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 3175A67<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * Replay_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
| | | * FIFO_0<br />
| | | * FIFO_1<br />
<br />
<br />
</pre><br />
<br />
<br />
====N321====<br />
<br />
<pre><br />
$ uhd_usrp_probe<br />
[INFO] [UHD] linux; GNU C++ version 7.3.1 20180712 (Red Hat 7.3.1-6); Boost_106400; UHD_3.14.0.0-0-g6875d061<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.20.2,type=n3xx,product=n320,serial=3166646,claimed=False,addr=192.168.20.2<br />
[INFO] [MPM.PeriphManager] init() called with device args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'.<br />
[INFO] [MPM.Rhodium-0] init() called with args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'<br />
[INFO] [MPM.Rhodium-1] init() called with args `time_source=internal,clock_source=internal,product=n320,mgmt_addr=192.168.20.2'<br />
[INFO] [0/Replay_0] Initializing block control (NOC ID: 0x4E91A00000000004)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000320)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/FIFO_0] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
[INFO] [0/FIFO_1] Initializing block control (NOC ID: 0xF1F0000000000000)<br />
_____________________________________________________<br />
/<br />
| Device: N300-Series Device<br />
| _____________________________________________________<br />
| /<br />
| | Mboard: ni-n3xx-3166646<br />
| | eeprom_version: 2<br />
| | mpm_version: 3.14.0.0-g6875d061<br />
| | pid: 16962<br />
| | product: n320<br />
| | rev: 6<br />
| | rpc_connection: remote<br />
| | serial: 3166646<br />
| | type: n3xx<br />
| | MPM Version: 1.2<br />
| | FPGA Version: 5.3<br />
| | FPGA git hash: 3de8954.clean<br />
| | RFNoC capable: Yes<br />
| | <br />
| | Time sources: internal, external, gpsdo, sfp0<br />
| | Clock sources: external, internal, gpsdo<br />
| | Sensors: gps_sky, gps_time, gps_gpgga, gps_locked, fan, gps_tpv, ref_locked, temp<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D814<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: B<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D810<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, RX2, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | RX Codec: A<br />
| | | | Name: ad9695-625<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: B<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D814<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: B<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | TX Dboard: A<br />
| | | ID: Unknown (0x0152)<br />
| | | Serial: 316D810<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Frontend: 0<br />
| | | | Name: Rhodium<br />
| | | | Antennas: TX/RX, CAL, TERM<br />
| | | | Sensors: lo_locked<br />
| | | | Freq range: 1.000 to 6000.000 MHz<br />
| | | | Gain range all: 0.0 to 60.0 step 1.0 dB<br />
| | | | Bandwidth range: 250000000.0 to 250000000.0 step 0.0 Hz<br />
| | | | Connection Type: <br />
| | | | Uses LO offset: No<br />
| | | _____________________________________________________<br />
| | | /<br />
| | | | TX Codec: A<br />
| | | | Name: dac37j82<br />
| | | | Gain Elements: None<br />
| | _____________________________________________________<br />
| | /<br />
| | | RFNoC blocks on this device:<br />
| | | <br />
| | | * Replay_0<br />
| | | * Radio_0<br />
| | | * Radio_1<br />
| | | * DDC_0<br />
| | | * DDC_1<br />
| | | * DUC_0<br />
| | | * DUC_1<br />
| | | * FIFO_0<br />
| | | * FIFO_1<br />
<br />
</pre><br />
<br />
If you see warnings such as:<br />
<br />
[WARNING] [UDP] The recv buffer could not be resized sufficiently.<br />
<br />
You need to resize the socket buffers for your network interface card:<br />
<br />
sudo sysctl -w net.core.rmem_max=288000<br />
sudo sysctl -w net.core.wmem_max=288000<br />
sudo sysctl -w net.core.rmem_max=33554432<br />
<br />
===ASCII Art Example===<br />
The UHD driver includes several example programs, which may serve as test programs or the basis for your application program. The source code can be obtained from the UHD repository on github at: https://github.com/EttusResearch/uhd/tree/master/host/examples<br />
<br />
You can quickly verify the operation of your USRP N3xx by running the <code>rx_ascii_art_dft</code> UHD example program. <br />
<br />
The <code>rx_ascii_art_dft</code> utility is a simple console based, real-time FFT display tool. It is not graphical in nature, so it can be easily run over an SSH connection within a terminal window, and does not need any graphical capability, such as X Windows, to be installed. It can also be run over a serial console connection, although this is not recommended, as the formatting may not render correctly.<br />
<br />
You can run a simple test of the N3xx USRP by connecting an antenna and observing the spectrum of a commercial FM radio station in real-time, following the steps below:<br />
<br />
1. Attach an antenna to the <code>Ch0/RX2</code> antenna port of the N3xx.<br />
<br />
2. From your host computer, run the command:<br />
<br />
'''N300/N310'''<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
</pre><br />
<br />
'''N320/N321'''<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=250e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
</pre><br />
<br />
NOTE: Modify the command line argument <code>freq</code> above to specify a tuning frequency for a strong local FM radio station. You will also need to update the IP Address to match your device IP.<br />
<br />
3. You should see a real-time FFT display of 2.5 MHz of spectrum, centered at the specified tuning frequency.<br />
<br />
4. Type "<code>Q</code>" or <code>Ctrl-C</code> to stop the program and to return to the Linux command line.<br />
<br />
5. You can run with the <code>--help</code> argument to see a description of all available command-line options.<br />
<br />
Example Output:<br />
<br />
<pre><br />
$ /usr/local/lib/uhd/examples/rx_ascii_art_dft --args "master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2" --freq 98.5e6 --rate 2.5e6 --gain 50 --ref-lvl="-50" --dyn-rng 90 --ant "RX2" --subdev "A:0"<br />
<br />
Creating the usrp device with: master_clock_rate=125e6,mgmt_addr=192.168.1.151,addr=192.168.10.2...<br />
[INFO] [UHD] linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_3.11.1.HEAD-0-gad6b0935<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.151,type=n3xx,product=n310,serial=313ABDA,claimed=False,master_clock_rate=125e6,addr=192.168.10.2<br />
[INFO] [MPM.main] Launching USRP/MPM, version: 3.11.1.0-gunknown<br />
[INFO] [MPM.main] Spawning RPC process...<br />
[INFO] [MPM.PeriphManager] Device serial number: 313ABDA<br />
[INFO] [MPM.PeriphManager] Found 2 daughterboard(s).<br />
[INFO] [MPM.RPCServer] RPC server ready!<br />
[INFO] [MPM.RPCServer] Spawning watchdog task...<br />
[INFO] [MPM.PeriphManager] init() called with device args `mgmt_addr=192.168.1.151,product=n310,master_clock_rate=125e6'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1336 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1338 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1346 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1350 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_2] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/Radio_3] Initializing block control (NOC ID: 0x12AD100000000310)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_2] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DDC_3] Initializing block control (NOC ID: 0xDDC0000000000001)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_2] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
[INFO] [0/DUC_3] Initializing block control (NOC ID: 0xD0C0000000000000)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
TX Channel: 2<br />
TX DSP: 0<br />
TX Dboard: C<br />
TX Subdev: Magnesium<br />
TX Channel: 3<br />
TX DSP: 0<br />
TX Dboard: D<br />
TX Subdev: Magnesium<br />
<br />
Setting RX Rate: 2.500000 Msps...<br />
Actual RX Rate: 2.500000 Msps...<br />
<br />
Setting RX Freq: 98.500000 MHz...<br />
Actual RX Freq: 98.500000 MHz...<br />
<br />
Setting RX Gain: 50.000000 dB...<br />
Actual RX Gain: 50.000000 dB...<br />
<br />
Checking RX: all_los: locked ...<br />
<br />
Done!<br />
</pre><br />
<br />
===Benchmarking your system===<br />
Included with the UHD driver example programs is a utility, <code>benchmark_rate</code> to benchmark the transport link of the system. <br />
<br />
A system's maximum performance is dependent upon many factors. <code>benchmark_rate</code> will exercise the transport link and CPU of the system. <br />
<br />
====1 Gb Interface====<br />
NOTE: This example requires the <code>HG</code> FPGA image to be loaded.<br />
<br />
'''N300/N310'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 3.84 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=122.88e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 3.84e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 3.84e6 \<br />
--tx_subdev "A:0"<br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 1.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 1.25e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 1.25e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
'''N320/N321'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 3.84 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,master_clock_rate=245.76e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 3.84e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 3.84e6 \<br />
--tx_subdev "A:0"<br />
<br />
When streaming samples over a 1 Gb transport link, the maximum accumulative rate for all channels is 25 MS/s with a <code>sc16</code> OTW format. To achieve higher streaming rates, it is recommended to use the 10 Gb interfaces.<br />
<br />
====10 Gb Interface SFP 1====<br />
NOTE: This example will work with either the <code>HG</code> or <code>XG</code> FPGA image.<br />
<br />
'''N300/N310'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 31.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 31.25e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 31.25e6 \<br />
--tx_subdev "A:0" <br />
<br />
'''N320/N321'''<br />
<br />
This example will test one full-duplex stream using "RF0/A:0", at a rate of 31.25 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=250e6" \<br />
--duration 60 \<br />
--channels "0" \<br />
--rx_rate 31.25e6 \<br />
--rx_subdev "A:0" \<br />
--tx_rate 31.25e6 \<br />
--tx_subdev "A:0" <br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 30.72 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=122.88e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 30.72e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 30.72e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
'''N320/N321'''<br />
<br />
This example will test two full-duplex streams at 30.72 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.20.2,master_clock_rate=245.76e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 30.72e6 \<br />
--rx_subdev "A:0 B:0" \<br />
--tx_rate 30.72e6 \<br />
--tx_subdev "A:0 B:0"<br />
<br />
====Dual 10 Gb Interface====<br />
NOTE: This example requires the <code>XG</code> FPGA image to be loaded.<br />
<br />
'''N310'''<br />
<br />
This example will test four full-duplex streams at 62.5 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 62.5e6 \<br />
--rx_subdev "A:0 A:1 B:0 B:1" \<br />
--tx_rate 62.5e6 \<br />
--tx_subdev "A:0 A:1 B:0 B:1"<br />
<br />
<br />
'''N320/N321'''<br />
<br />
This example will test two full-duplex streams at 62.5 MS/s, for 60 seconds:<br />
<br />
/usr/local/lib/uhd/examples/benchmark_rate \<br />
--args "type=n3xx,mgmt_addr=192.168.1.151,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=250e6" \<br />
--duration 60 \<br />
--channels "0,1,2,3" \<br />
--rx_rate 62.5e6 \<br />
--rx_subdev "A:0 B:0" \<br />
--tx_rate 62.5e6 \<br />
--tx_subdev "A:0 B:0"<br />
<br />
==USRP N3xx Device Specific Operations==<br />
<br />
===White Rabbit Ethernet-Based Synchronization===<br />
* [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]]<br />
<br />
<br />
===N320/N321===<br />
* [[USRP N320/N321 LO Distribution]]<br />
* [[5G NR EVM Measurements with the USRP N320/N321]]<br />
<br />
<br />
===Turning the Device Off/On===<br />
To avoid damaging the file system and causing any corruption, do not turn the device off with the power button without first shutting down the system. Use this command to cleanly and properly shut the system down:<br />
<br />
shutdown -h now<br />
<br />
===Enable Auto Booting===<br />
Auto booting of the N3xx when power is applied can be configured by enabling the flag on the device's EEPROM with the following command:<br />
<br />
eeprom-set-flags 0x1<br />
<br />
===Default Password===<br />
The default user is <code>root</code> and the password is empty (no password).<br />
<br />
It is recommended to update the <code>root</code> password, which can be done with the command <code>passwd</code>:<br />
<br />
Example Output:<br />
<br />
root@ni-n3xx-serial:~# passwd<br />
Changing password for root<br />
New password: <br />
Re-enter new password: <br />
passwd: password changed.<br />
<br />
==Technical Support and Community Knowledge Base==<br />
Technical support for USRP hardware is available through email only. If the product arrived in a nonfunctional state or you require technical assistance, please contact [mailto:support@ettus.com support@ettus.com]. Please allow 24 to 48 hours for response by email, depending on holidays and weekends, although we are often able to reply more quickly than that.<br />
<br />
We also recommend that you subscribe to the community mailing lists. The mailing lists have a responsive and knowledgeable community of hundreds of developers and technical users who are located around the world. When you join the community, you will be connected to this group of people who can help you learn about SDR and respond to your technical and specific questions. Often your question can be answered quickly on the mailing lists. Each mailing list also provides an archive of all past conversations and discussions going back many years. Your question or problem may have already been addressed before, and a relevant or helpful solution may already exist in the archive.<br />
<br />
Discussions involving the USRP hardware and the UHD software itself are best addressed through the '''usrp-users''' mailing list at [http://usrp-users.ettus.com http://usrp-users.ettus.com].<br />
<br />
Discussions involving the use of [http://gnuradio.org/ GNU Radio] with USRP hardware and UHD software are best addressed through the '''discuss-gnuradio''' mailing list at [https://lists.gnu.org/mailman/listinfo/discussgnuradio https://lists.gnu.org/mailman/listinfo/discussgnuradio].<br />
<br />
Discussions involving the use of [http://openbts.org/ OpenBTS®] with USRP hardware and UHD software are best addressed through the '''openbts-discuss''' mailing list at [https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss].<br />
<br />
The support page on our website is located at [https://www.ettus.com/support https://www.ettus.com/support]. The Knowledge Base is located at [https://kb.ettus.com https://kb.ettus.com].<br />
<br />
==Legal Considerations==<br />
Every country has laws governing the transmission and reception of radio signals. Users are solely responsible for insuring they use their USRP system in compliance with all applicable laws and regulations. Before attempting to transmit and/or receive on any frequency, we recommend that you determine what licenses may be required and what restrictions may apply.<br />
<br />
*NOTE: This USRP product is a piece of test equipment.<br />
<br />
==Sales and Ordering Support==<br />
If you have any non-technical questions related to your order, then please contact us by email at [mailto:orders@ettus.com orders@ettus.com], or by phone at +14086106399 (Monday-Friday, 8 AM - 5 PM, Pacific Time). Please be sure to include your order number and the serial number of your USRP.<br />
<br />
==Terms and Conditions of Sale==<br />
Terms and conditions of sale can be accessed online at the following link: http://www.ettus.com/legal/terms-and-conditions-of-sale<br />
<br />
[[Category:Getting Started Guides]]<br />
[[Category:N300]]<br />
[[Category:N310]]<br />
<br />
[[Category:N320]]<br />
<br />
[[Category:N321]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6014Getting Started with DPDK and UHD2024-01-30T23:09:10Z<p>MichaelDickens: /* Limit Execution CPUs/Cores */ add DPDK info</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
** https://doc.dpdk.org/guides-23.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
NOTE: For best performance do ''not'' include the cores used by DPDK in the <code>dpdk_corelist</code> argument in the <code>taskset</code>.<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd containerd.socket<br />
systemctl stop docker docker.socket<br />
systemctl stop dbus dbus.socket<br />
systemctl stop snapd snapd.socket<br />
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6013Getting Started with DPDK and UHD2024-01-30T23:05:18Z<p>MichaelDickens: /* Disable System Interrupts on Cores/CPUs */ add DPDK info</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
** https://doc.dpdk.org/guides-23.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd containerd.socket<br />
systemctl stop docker docker.socket<br />
systemctl stop dbus dbus.socket<br />
systemctl stop snapd snapd.socket<br />
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6012Getting Started with DPDK and UHD2024-01-30T23:04:57Z<p>MichaelDickens: /* Isolate Cores/CPUs */ add DPDK info</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
** https://doc.dpdk.org/guides-23.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect. It is OK to isolate core used by DPDK via this method.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd containerd.socket<br />
systemctl stop docker docker.socket<br />
systemctl stop dbus dbus.socket<br />
systemctl stop snapd snapd.socket<br />
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6011Getting Started with DPDK and UHD2024-01-25T16:05:23Z<p>MichaelDickens: /* Stopping Extraneous Processes */ remove extra</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
** https://doc.dpdk.org/guides-23.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd containerd.socket<br />
systemctl stop docker docker.socket<br />
systemctl stop dbus dbus.socket<br />
systemctl stop snapd snapd.socket<br />
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6010Getting Started with DPDK and UHD2024-01-25T16:03:46Z<p>MichaelDickens: /* Stopping Extraneous Processes */ reorganize and add udev*</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
** https://doc.dpdk.org/guides-23.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd containerd.socket<br />
systemctl stop docker docker.socket<br />
systemctl stop dbus dbus.socket<br />
systemctl stop snapd snapd.socket<br />
systemctl stop snapd<br />
systemctl stop udev systemd-udevd-control.socket systemd-udevd-kernel.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6009Getting Started with DPDK and UHD2024-01-24T21:37:38Z<p>MichaelDickens: /* Stopping Extraneous Processes */ add dbus</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
** https://doc.dpdk.org/guides-23.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop dbus<br />
systemctl stop snapd<br />
systemctl stop containerd.socket<br />
systemctl stop docker.socket<br />
systemctl stop dbus.socket<br />
systemctl stop snapd.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6008Getting Started with DPDK and UHD2024-01-19T20:59:23Z<p>MichaelDickens: /* References */ ad 23.11</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
** https://doc.dpdk.org/guides-23.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
systemctl stop containerd.socket<br />
systemctl stop docker.socket<br />
systemctl stop snapd.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6007Getting Started with DPDK and UHD2024-01-19T20:58:51Z<p>MichaelDickens: /* Installing DPDK */ fix DPDK doc links</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-19.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-20.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-22.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
* [https://doc.dpdk.org/guides-23.11/linux_gsg/build_dpdk.html DPDK 23.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
systemctl stop containerd.socket<br />
systemctl stop docker.socket<br />
systemctl stop snapd.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=USRP_X410/X440_Getting_Started_Guide&diff=6006USRP X410/X440 Getting Started Guide2024-01-17T17:57:38Z<p>MichaelDickens: /* FPGA Image Flavors */ whitespace</p>
<hr />
<div>==Kit Contents==<br />
===X4x0===<br />
{|<br />
|style="vertical-align:top"|<br />
* NI Ettus USRP X410 or X440<br />
* DC Power Supply (12V, 20A)<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to USB-C Cable (1m)<br />
* Getting Started Guide URL (QR Code)<br />
* Safety, Environmental, and Regulatory Information<br />
||[[File:X410.jpg|450px|center]]<br />
||[[File:X440.jpg|450px|center]]<br />
|}<br />
<br />
==USRP X440 Design Considerations==<br />
* https://kb.ettus.com/About_Sampling_Rates_and_Master_Clock_Rates_for_the_USRP_X440<br />
<br />
==You Will Need==<br />
* For Network Mode: A host computer with an available 1 or 10 Gigabit Ethernet interface for sample streaming. In addition to the Ethernet interface used for sampling streaming, your host computer will require a separate 1 Gigabit Ethernet interface for command and control streaming.<br />
<br />
* For Stand-Alone Embedded Mode: A host computer with an available 1 Gigabit Ethernet port or a USB 2.0 port to remotely access the embedded Linux operating system running on ARM CPU.<br />
<br />
==Proper Care and Handling==<br />
<br />
All Ettus Research products are individually tested before shipment. The USRP is guaranteed to be functional at the time it is received by the customer. Improper use or handling of the USRP can cause the device to become non-functional. Take the following precautions to prevent damage to the unit.<br />
<br />
* Never allow metal objects to touch the circuit board while powered.<br />
* Always properly terminate the transmit port with an antenna or 50Ω load.<br />
* Always handle the board with proper anti-static methods.<br />
* Never allow the board to directly or indirectly come into contact with any voltage spikes.<br />
* Never allow any water or condensing moisture to come into contact with the device.<br />
* Always use caution with FPGA, firmware, or software modifications.<br />
<br />
{|<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |X410: Never apply more than +14 dBm continuous <=3GHz, +17 dBm continuous >3GHz, or +20dBm more than 5 minutes >3GHz of power into any RF input.<br />
|-<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |X440: Never apply more than +13 dBm continuous <=2.5GHz, +17 dBm continuous between 2.5GHz and 3.6 GHz, or +20dBm continuous between 3.6 GHz and 4 GHz of power into any RF input.<br />
|-<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |X440: Always use at least 30dB attenuation if operating in loopback configuration<br />
|-<br />
|}<br />
<br />
==Install and Setup the Software Tools on Your Host Computer==<br />
In order to use your Universal Software Radio Peripheral (USRP™), you must have the software tools correctly installed and configured on your host computer. The easiest way to install USRP Hardware Driver (UHD) is by getting a binary installer package for your operating system as described in the UHD manual about [https://files.ettus.com/manual/page_install.html Binary Installation]. If no binary packages are available for your operating system or you want to modify the sources by yourself, a step-by-step guide is available at the Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux|Linux]], [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X|OS X]] and [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows|Windows]] Application Notes.<br />
<br />
To find the latest release of UHD, see the UHD repository at https://github.com/EttusResearch/uhd.<br />
<br />
The USRP X410 requires UHD version 4.1 or later.<br />
The USRP X440 requires UHD version 4.5 or later. <br />
<br />
'''When you receive a brand-new device, it is strongly recommended that you download the latest filesystem image from the Ettus Research website update the unit. It is not recommended that you use the filesystem from the factory as-is. Instructions on downloading the latest filesystem image and updating it is listed below.'''<br />
<br />
'''Note that if you are operating the device in Network Mode, the version of UHD running on the host computer and the USRP X4x0 must match.'''<br />
<br />
==Assembling the X4x0==<br />
Inside the kit you will find the X4x0 and an X4x0 power supply. Plug these in, connect the 1GbE RJ45 interface to your network, and power on the device by pressing the power button.<br />
<br />
<br />
==The STM32 Microcontroller==<br />
<br />
The STM32 microcontroller (also referred to as the "SCU") controls various low-level features of the X4x0 series motherboard: It controls the power sequencing, reads out fan speeds and some of the temperature sensors. It is connected to the RFSoC via an I2C bus. It is running software based on Chromium EC.<br />
<br />
It is possible to log into the STM32 using the serial interface (see Connecting to the Microcontroller). This will allow certain low-level controls, such as remote power cycling should the CPU have become unresponsive for whatever reason.<br />
<br />
===Updating the SCU===<br />
<br />
The writable SCU image file is stored on the filesystem under /lib/firmware/ni/ec-titanium-revX.RW.bin (where X is a revision compatibility number). To update, simply replace the .bin file with the updated version and reboot.<br />
<br />
==eMMC Storage==<br />
<br />
The main non-volatile storage of the USRP is a 16 GB eMMC storage. This storage can be made accessible as a USB Mass Storage device through the USB-OTG connector on the back panel.<br />
<br />
The entire root file system (Linux kernel, libraries) and any user data are stored on the eMMC. It is partitioned into four partitions:<br />
<br />
Boot partition (contains the bootloader). This partition usually does not require modification.<br />
A data partition, mounted in /data. This is the only partition that is not erased during file system updates.<br />
Two identical system partitions (root file systems). These contain the operating system and the home directory (anything mounted under / that is not the data or boot partition). The reason there are two of these is to enable remote updates: An update running on one partition can update the other one without any effect to the currently running system. Note that the system partitions are erased during updates and are thus unsuitable for permanently storing information.<br />
Note: It is possible to access the currently inactive root file system by mounting it. After logging into the device using serial console or SSH (see the following two sections), run the following commands:<br />
<br />
<pre><br />
$ mkdir temp<br />
<br />
$ mount /dev/mmcblk0p3 temp # This assumes mmcblk0p3 is currently not mounted<br />
<br />
$ ls temp # You are now accessing the idle partition:<br />
<br />
bin data etc lib media proc sbin tmp usr<br />
boot dev home lost+found mnt run sys uboot var<br />
</pre><br />
<br />
The device node in the mount command might differ, depending on which partition is currently already mounted.<br />
<br />
==USB Access to eMMC==<br />
<br />
While Mender should be used for routine filesystem updates (see Updating Filesystems), it is also possible to access the X4x0's internal eMMC from an external host over USB. This allows accessing or modifying the filesystem, as well as the ability to flash the device with an entirely new filesystem.<br />
<br />
In order to do so, you'll need an external computer with two USB ports, and two USB cables to connect the computer to your X4x0. The instructions below assume a Linux host.<br />
<br />
First, connect to the APU serial console at a baud rate of 115200. Boot the device, and stop the boot sequence by typing noautoboot at the prompt. Then, run the following command in the U-boot command prompt:<br />
<br />
<code>ums 0 mmc 0</code><br />
<br />
This will start the USB mass storage gadget to expose the eMMC as a USB mass storage device. You should see a spinning indicator on the console, which indicates the gadget is active.<br />
<br />
Next, connect your external computer to the X4x0's USB to PS port using an OTG cable. Your computer should recognize the X4x0 as a mass storage device, and you should see an entry in your kernel logs (dmesg) that looks like this:<br />
<br />
<pre><br />
usb 3-1: New USB device found, idVendor=3923, idProduct=7a7d, bcdDevice= 2.23<br />
usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0<br />
usb 3-1: Product: USB download gadget<br />
usb 3-1: Manufacturer: National Instruments<br />
sd 6:0:0:0: [sdc] 30932992 512-byte logical blocks: (15.8 GB/14.8 GiB)<br />
sdc: sdc1 sdc2 sdc3 sdc4<br />
sd 6:0:0:0: [sdc] Attached SCSI removable disk<br />
</pre><br />
<br />
The exact output will depend on your machine, but from this log you can see that the X4x0 was recognized and /dev/sdc is the block device representing the eMMC, with 4 partitions detected (see eMMC Storage for details on the partition layout).<br />
<br />
It is now possible to treat the X4x0's eMMC as you would any other USB drive: the individual partitions can be mounted and accessed, or the entire block device can be read/written.<br />
<br />
Once you're finished accessing the device over USB, the u-boot gadget may be stopped by hitting Ctrl-C at the APU serial console.<br />
<br />
<br />
== Flashing the eMMC ==<br />
<br />
Once the X4x0's eMMC is accessible over USB, it's possible to write the filesystem image and thus change the device's filesystem. You can obtain the latest filesystem image by running:<br />
<br />
<code>uhd_images_downloader -t sdimg -t x4xx</code><br />
<br />
The output of this command will indicate where the downloaded images were put, or specify a custom location using using the <code>-i INSTALL_LOCATION</code> argument.<br />
<br />
There are 2 ways to write the image to the X4x0's eMMC: using <code>dd</code> and <code>bmaptool</code>. Run one of the following commands, replacing <code>/dev/sdX</code> with the block device of the X4x0's eMMC (found in the device's kernel log or by running <code>lsblk</code>). Take care to use the correct block device or else you might overwrite the wrong drive!<br />
<br />
<code>sudo dd if=/path/to/usrp_x4xx_fs.sdimg of=/dev/sdX bs=1M</code><br />
<br />
<code>sudo bmaptool copy --bmap /path/to/usrp_x4xx_fs.sdimg.bmap /path/to/usrp_x4xx_fs.sdimg /dev/sdX</code><br />
<br />
The former is generally preferred as it will always work, even if it slower than the latter.<br />
<br />
==Using a USRP X4x0 from UHD==<br />
Like any other USRP, all X4x0 USRPs are controlled by the UHD software. To integrate a USRP X4x0 into your C++ application, you would generate a UHD device in the same way you would for any other USRP:<br />
<br />
<code>auto usrp = uhd::usrp::multi_usrp::make("type=x4xx");</code><br />
<br />
For a list of which arguments can be passed into make(), see Section Device Arguments.<br />
<br />
==Updating Filesystems==<br />
<br />
Mender is a third-party software that enables remote updating of the root file system without physically accessing the device (see also the [https://mender.io/ Mender website]). Mender can be executed locally on the device, or a Mender server can be set up which can be used to remotely update an arbitrary number of USRP devices. Mender servers can be self-hosted, or hosted by Mender (see mender.io for pricing and availability).<br />
<br />
When updating the file system using Mender, the tool will overwrite the root file system partition that is not currently mounted (note: the onboard flash storage contains two separate root file system partitions, only one is ever used at a single time). Any data stored on that partition will be permanently lost, including the currently loaded FPGA image. After updating that partition, it will reboot into the newly updated partition. Only if the update is confirmed by the user, the update will be made permanent. This means that if an update fails, the device will be always able to reboot into the partition from which the update was originally launched (which presumably is in a working state). Another update can be launched now to correct the previous, failed update, until it works.<br />
<br />
To obtain the file system Mender image (these are files with a <code>.mender</code> suffix), run the following command on the host computer with Internet access:<br />
<br />
$ sudo uhd_images_downloader -t mender -t x4xx --yes<br />
<br />
NOTE: In the output of the command, the folder destination where the images are saved is printed out.<br />
<br />
Next, you will need to copy this Mender file system image to the USRP X4xx. This can be done with the Linux utility <code>scp</code>.<br />
<br />
$ scp /usr/local/share/uhd/images/usrp_x4xx_fs.mender root@192.168.1.51:~/. <br />
<br />
Note: The path and IP may different for your configuration, the command above assumes you're using the default installation path of <code>/usr/local</code> and that the X4xx's IP is <code>192.168.1.51</code>.<br />
<br />
After copying the Mender file system image to the X4xx, connect to the X4xx using either the Serial Console, or via SSH to gain shell access.<br />
<br />
On the X4xx, run <code>mender install /path/to/latest.mender</code> to update the file system:<br />
<br />
$ mender install /home/root/usrp_x4xx_fs.mender<br />
<br />
The artifact can also be stored on a remote server:<br />
$ mender install <nowiki>http://server.name/path/to/latest.mender</nowiki><br />
<br />
This procedure will take a few minutes to complete. After mender has logged a successful update, reboot the device:<br />
$ reboot<br />
<br />
If the reboot worked, and the device seems functional, commit the changes so that the boot loader knows to permanently boot into this partition:<br />
$ mender -commit<br />
<br />
To identify the currently installed Mender artifact from the command line, the following file can be queried on the X4x0:<br />
$ cat /etc/mender/artifact_info<br />
<br />
If you are using a Mender server, the updates can be initiated from a web dashboard. From there, you can start the updates without having to log into the device, and you can update groups of USRPs with a few clicks in a web GUI. The dashboard can also be used to inspect the state of USRPs. This is a simple way to update groups of rack-mounted USRPs with custom file systems.<br />
<br />
If you are running a hosted server, the updates can be initiated from a web dashboard. From there, you can start the updates without having to log into the device, and can update groups of USRPs with a few clicks in a web GUI. The dashboard can also be used to inspect the state of USRPs. This is a simple way to update groups of rack-mounted USRPs with custom file systems.<br />
<br />
==Network Interfaces==<br />
The Ettus USRP X4x0 has various network interfaces:<br />
<br />
eth0: RJ45 port.<br />
<br />
The RJ45 port comes up with a default configuration of DHCP, that will request a network address from your DHCP server (if available on your network). This interface is agnostic of FPGA image flavor.<br />
<br />
int0: internal interface for network communication between the embedded ARM processor and FPGA.<br />
<br />
The internal network interface is configured with a static address: 169.254.0.1/24. This interface is agnostic of FPGA image flavor.<br />
<br />
sfpX [, sfpX_1, sfpX_2, sfpX_3]: QSFP28 network interface(s), up-to four (one per lane) based on implemented protocol.<br />
<br />
Each QSFP28 port has four high-speed transceiver lanes. Therefore, depending on the FPGA image flavor, up-to four different network interfaces may exist per QSFP28 port, using the sfpXfor the first lane, and sfpX_1-3 for the other three lanes. Each network interface has a default static IP address. Note that for multi-lane protocols, such as 100 GbE, a single interface is used (sfpX).<br />
The configuration files for these network interfaces are stored in: <code>/data/network/</code><br />
<br />
{| class="wikitable" <br />
|-<br />
! Interface Name<br />
! Description<br />
! Default Configuration<br />
! Configuration File<br />
! Example: X4_200/X4_400 FPGA image<br />
|-<br />
| eth0<br />
| RJ45<br />
| style="vertical-align:middle; background-color:#FFF;" | DHCP<br />
| style="vertical-align:middle; background-color:#FFF;" | eth0.network<br />
| DHCP<br />
|-<br />
| int0<br />
| Internal<br />
| style="vertical-align:middle; background-color:#FFF;" | 169.254.0.1/24<br />
| style="vertical-align:middle; background-color:#FFF;" | int0.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 169.254.0.1/24<br />
|-<br />
| sfp0<br />
| QSFP28 0 (4-lanes interface or lane 0)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.10.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp0.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.10.2/24<br />
|-<br />
| style="background-color:#FFF;" | sfp0_1<br />
| QSFP28 0 (lane 1)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.11.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp0_1.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.11.2/24<br />
|-<br />
| style="background-color:#FFF;" | sfp0_2<br />
| QSFP28 0 (lane 2)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.12.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp0_2.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.12.2/24<br />
|-<br />
| sfp0_3<br />
| QSFP28 0 (lane 3)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.13.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp0_3.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.13.2/24<br />
|-<br />
| sfp1<br />
| QSFP28 1 (4-lanes interface or lane 0)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.20.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp1.network<br />
| style="vertical-align:middle; background-color:#FFF;" | N/C<br />
|-<br />
| sfp1_1<br />
| QSFP28 1 (lane 1)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.21.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp1_1.network<br />
| style="vertical-align:middle; background-color:#FFF;" | N/C<br />
|-<br />
| style="background-color:#FFF;" | sfp1_2<br />
| QSFP28 1 (lane 2)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.22.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp1_2.network<br />
| style="vertical-align:middle; background-color:#FFF;" | N/C<br />
|-<br />
| style="background-color:#FFF;" | sfp1_3<br />
| QSFP28 1 (lane 3)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.23.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp1_3.network<br />
| style="vertical-align:middle; background-color:#FFF;" | N/C<br />
|-<br />
|}<br />
<br />
==Network Connectivity==<br />
Once the X4x0 has booted, determine the IP address and verify network connectivity by running uhd_find_devices on the host computer:<br />
<br />
X410:<br />
<pre><br />
$ uhd_find_devices<br />
<br />
-- UHD Device 0<br />
<br />
Device Address:<br />
serial: 1234ABC<br />
addr: 10.2.161.10<br />
claimed: False<br />
mgmt_addr: 10.2.161.10<br />
product: x410<br />
type: x4xx<br />
</pre><br />
<br />
X440:<br />
<pre><br />
$ uhd_find_devices<br />
<br />
-- UHD Device 0<br />
<br />
Device Address:<br />
serial: 1234ABC<br />
addr: 10.2.161.10<br />
claimed: False<br />
mgmt_addr: 10.2.161.10<br />
product: x440<br />
type: x4xx<br />
</pre><br />
<br />
By default, an X4x0 will use DHCP to attempt to find an address.<br />
<br />
At this point, you should run:<br />
<br />
<code>uhd_usrp_probe --args addr=<IP address></code><br />
to ensure functionality of the device.<br />
<br />
Note: If you receive the following error:<br />
<br />
<code>Error: RuntimeError: Graph edge list is empty for rx channel 0</code><br />
then you will need to download a UHD-compatible FPGA as described in Updating the FPGA or using the following command (it assumes that FPGA images have been downloaded previously using uhd_images_downloader, or that the command is run on the device itself):<br />
<br />
X410:<br />
<code>uhd_image_loader --args type=x4xx,addr=<ip address>,fpga=X4_200</code><br />
<br />
X440:<br />
<code>uhd_image_loader --args type=x4xx,addr=<ip address>,fpga=X4_400</code><br />
<br />
When running on the device, use <code>127.0.0.1</code> as the IP address.<br />
<br />
You can now use existing UHD examples or applications (such as rx_sample_to_file, rx_ascii_art_dft, or tx_waveforms) or other UHD-compatible applications to start receiving and transmitting with the device.<br />
<br />
See Network Interfaces for further details on the various network interfaces available on the X4x0.<br />
<br />
<br />
===Network Status LEDs===<br />
The Ettus USRP X4x0 is equipped with status LEDs for its network-capable ports: RJ45 and QSFP28s, see RJ45 LED Behavior and QSFP28 LED Behavior accordingly.<br />
<br />
====RJ45 LED Behavior====<br />
The RJ45 port has two independent LEDs: green (right) and yellow (left). The table below summarizes the LEDs' behavior. Note that link speed indication is not currently supported.<br />
<br />
{| class="wikitable" <br />
|- style="font-weight:bold; text-align:center; vertical-align:middle;"<br />
! Link / Activity<br />
! Green LED<br />
! Yellow LED<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | No Link<br />
| Off<br />
| Off<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | Link / No Activity<br />
| On<br />
| Off<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | Link / Activity<br />
| On<br />
| Blinking<br />
|}<br />
<br />
====QSFP28 LED Behavior====<br />
Each QSFP28 connector has four LEDs, one for each high-speed transceiver lane. The table below summarizes the LEDs' behavior, note that for multi-lane protocols, such as 100 GbE, the corresponding LEDs are ganged together. Within the same image, multiple speeds on the same port (e.g., both 10 GbE and 100 GbE) are not supported, therefore link speed indication is not supported.<br />
<br />
{| class="wikitable" <br />
|- style="font-weight:bold; text-align:center; vertical-align:middle;"<br />
! Link / Activity<br />
! QSFP28 LED (4 Total)<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | No Link<br />
| Off<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | Link / No Activity<br />
| Green (solid)<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | Link / Activity<br />
| Amber (blinking)<br />
|}<br />
<br />
==Security-related Settings==<br />
The X4x0 ships without a root password set. It is possible to ssh into the device by simply connecting as root, and thus gaining access to all subsystems. To set a password, run the command<br />
<br />
<code>$ passwd</code><br />
on the device.<br />
<br />
==Serial Connection==<br />
It is possible to gain access to the device using a serial terminal emulator. To do so, the USB debug port needs to be connected to a separate computer to gain access. Most Linux, OSX, or other Unix flavors have a tool called 'screen' which can be used for this purpose, by running the following command:<br />
<br />
<code>$ sudo screen /dev/ttyUSB2 115200</code><br />
In this command, we prepend 'sudo' to elevate user privileges (by default, accessing serial ports is not available to regular users), we specify the device node (in this case, /dev/ttyUSB2), and the baud rate (115200).<br />
<br />
The exact device node depends on your operating system's driver and other USB devices that might be already connected. Modern Linux systems offer alternatives to simply trying device nodes; instead, the OS might have a directory of symlinks under /dev/serial/by-id:<br />
<br />
<pre>$ ls /dev/serial/by-id<br />
usb-Digilent_Digilent_USB_Device_2516351DDCC0-if02-port0<br />
usb-Digilent_Digilent_USB_Device_2516351DDCC0-if03-port0<br />
</pre><br />
<br />
Note: Exact names depend on the host operating system version and may differ.<br />
<br />
The first (with the if02 suffix) connects to the STM32 microcontroller (SCU), whereas the second (with the if03 suffix) connects to Linux running on the RFSoC APU.<br />
<br />
<code>$ sudo screen /dev/serial/by-id/usb-Digilent_Digilent_USB_Device_2516351DDCC0-if03-port0 115200</code><br />
After entering the username root (no password is set by default), you should be presented with a shell prompt similar to the following:<br />
<br />
<code>root@ni-x4xx-1234ABC:~#</code><br />
On this prompt, you can enter any Linux command available. Using the default configuration, the serial console will also show all kernel log messages (unlike when using SSH, for example), and give access to the boot loader (U-boot prompt). This can be used to debug kernel or bootloader issues more efficiently than when logged in via SSH.<br />
<br />
==Connecting to the Microcontroller==<br />
The microcontroller (which controls the power sequencing, among other things) also has a serial console available. To connect to the microcontroller, use the other UART device. In the example above:<br />
<br />
<code>$ sudo screen /dev/serial/by-id/usb-Digilent_Digilent_USB_Device_2516351DDCC0-if02-port0 115200</code><br />
<br />
It provides a very simple prompt. The command 'help' will list all available commands. A direct connection to the microcontroller can be used to hard-reset the device without physically accessing it and other low-level diagnostics. For example, running the command reboot will emulate a reset button press, resetting the state of the device, while the command powerbtn will emulate a power button press, turning the device back on again.<br />
<br />
==SSH Connection==<br />
The USRP X4x0 has two network connections: The dual QSFP28 ports, and an RJ45 connector. The latter is by default configured by DHCP; by plugging it into into 1 Gigabit switch on a DHCP-capable network, it will get assigned an IP address and thus be accessible via ssh.<br />
<br />
In case your network setup does not include a DHCP server, refer to the section Serial Connection. A serial login can be used to assign an IP address manually.<br />
<br />
After the device obtained an IP address you can log in from a Linux or OSX machine by typing:<br />
<br />
<code>$ ssh root@ni-x4xx-1234ABC # Replace with your actual device name!</code><br />
Depending on your network setup, using a .local domain may work:<br />
<br />
<code>$ ssh root@ni-x4xx-1234ABC.local</code><br />
Of course, you can also connect to the IP address directly if you know it (or set it manually using the serial console).<br />
<br />
Note: The device's hostname is derived from its serial number by default (<code>ni-x4xx-$SERIAL</code>). You can change the hostname by creating the file <code>/data/network/hostname</code>, saving the desired hostname in it, then rebooting.<br />
<br />
On Microsoft Windows, the connection can be established using a tool such as PuTTY, by selecting a username of root without password.<br />
<br />
Like with the serial console, you should be presented with a prompt like the following:<br />
<br />
<code>root@ni-x4xx-1234ABC:~#</code><br />
<br />
== Autoboot ==<br />
<br />
The USRP X4x0 can be configured to power on and boot automatically when power is applied. This setting can be controlled using the <code>eeprom-set-autoboot</code> script. This script is executed directly on the USRP X4x0. To enable autoboot, run <code>eeprom-set-autoboot on</code>; to disable autoboot, run <code>eeprom-set-autoboot off</code>.<br />
<br />
==Updating the FPGA==<br />
<br />
The FPGA can be updated simply using uhd_image_loader:<br />
<br />
<code>uhd_image_loader --args type=x4xx,addr=<IP address of device> --fpga-path <path to .bit></code><br />
or<br />
<br />
<code>uhd_image_loader --args type=x4xx,addr=<IP address of device>,fpga=FPGA_TYPE</code><br />
A UHD install will likely have pre-built images in /usr/share/uhd/images/. Up-to-date images can be downloaded using the uhd_images_downloader script:<br />
<br />
<code>uhd_images_downloader</code><br />
will download images into /usr/share/uhd/images/ (the path may differ, depending on how UHD was installed).<br />
<br />
Also note that the USRP already ships with compatible FPGA images on the device - these images can be loaded by SSH'ing into the device and running:<br />
<br />
X410:<br />
<code>uhd_image_loader --args type=x4xx,mgmt_addr=127.0.0.1,fpga=X4_200</code><br />
<br />
X440:<br />
<code>uhd_image_loader --args type=x4xx,mgmt_addr=127.0.0.1,fpga=X4_400</code><br />
<br />
==FPGA Image Flavors==<br />
Unlike the USRP X310 or other third-generation USRP devices, the FPGA image flavors do not only encode how the QSFP28 connectors are configured, but also which master clock rates are available. This is because the data converter configuration is part of the FPGA image (the ADCs/DACs on the X4x0 are on the same die as the FPGA). The image flavors consist of two short strings, separated by an underscore, e.g. X4_200 (X410) or X4_400 (X440) is an image flavor which contains 4x 10 GbE, and can handle an analog bandwidth of 200 MHz or 400 MHz respectively. The first two characters describe the configuration of the QSFP28 ports: 'X' stands for 10 GbE, 'C' stands for 100 GbE. For details see [https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_updating_fpga_types FPGA Image Flavor] in the [https://files.ettus.com/manual USRP Hardware Driver and USRP Manual].<br />
<br />
<br />
The analog bandwidth determines the available master clock rates. <br />
<br />
X410: As of UHD 4.1, only the X4_200 image is shipped with UHD, which allows a 245.76 MHz or 250 MHz master clock rate. With UHD 4.2, the CG_400 image was added allowing for 491.52 MHz and 500 MHz master clock rates. With UHD 4.5, the UC_200 image (245.76 MHz and 250 MHz master clock rate) was added.<br />
<br />
<br />
X440: As of UHD 4.5, UHD ships with X4_400, X4_1600, CG_400 and CG_1600 images. The X4_400 and CG_400 images allow master clock rates between 125 MHz and 512 MHz and the usage of all 8 channels while the X4_1600 and CG_1600 images allow master clock rates between 125 MHz and 2048 MHz but only the usage of channels 0 and 4.<br />
<br />
Any other images are considered experimental (unsupported).<br />
<br />
==Device Arguments==<br />
<br />
{| class="wikitable" style="vertical-align:middle;"<br />
|- style="font-weight:bold; text-align:center;"<br />
! Key<br />
! Description<br />
! Example Value<br />
|-<br />
| addr<br />
| IPv4 address of primary SFP+ port to connect to.<br />
| addr=192.168.30.2<br />
|-<br />
| second_addr<br />
| IPv4 address of secondary SFP+ port to connect to.<br />
| second_addr=192.168.40.2<br />
|-<br />
| mgmt_addr<br />
| IPv4 address or hostname to which to connect the RPC client. Defaults to `addr'.<br />
| mgmt_addr=ni-sulfur-311FE00<br />
|-<br />
| find_all<br />
| When using broadcast, find all devices, even if unreachable via CHDR.<br />
| find_all=1<br />
|-<br />
| master_clock_rate<br />
| Master Clock Rate in Hz.<br />
| master_clock_rate=250e6<br />
|-<br />
| converter_rate<br />
| Converter Rate in Hz. Only X440 and together with master_clock_rate.<br />
| master_clock_rate=250e6,converter_rate=1000e6<br />
|-<br />
| serialize_init<br />
| Force serial initialization of daughterboards.<br />
| serialize_init=1<br />
|-<br />
| skip_init<br />
| Skip the initialization process for the device.<br />
| skip_init=1<br />
|-<br />
| time_source<br />
| Specify the time (PPS) source.<br />
| time_source=internal<br />
|-<br />
| clock_source<br />
| Specify the reference clock source.<br />
| clock_source=internal<br />
|-<br />
| ref_clk_freq<br />
| Specify the external reference clock frequency, default is 10 MHz.<br />
| ref_clk_freq=20e6<br />
|-<br />
| discovery_port<br />
| Override default value for MPM discovery port.<br />
| discovery_port=49700<br />
|-<br />
| rpc_port<br />
| Override default value for MPM RPC port.<br />
| rpc_port=49701<br />
|}<br />
<br />
This is only a subset of the existing device arguments. For a complete list please consult the [https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_args UHD user manual of the X4x0 device series]. <br />
<br />
==GPS==<br />
<br />
The USRP X4x0 includes a Jackson Labs LTE-Lite GPS module. Its antenna port is on the rear panel (see Front and Back Panels). When the X4x0 has access to GPS satellite signals, it can use this module to read out the current GPS time and location as well as to discipline an onboard OCXO.<br />
<br />
To use the GPS as a clock and time reference, simply use gpsdo as a clock or time source. Alternatively, set gpsdo as a synchronization source:<br />
<br />
<pre><br />
// Set clock/time individually:<br />
usrp->set_clock_source("gpsdo");<br />
usrp->set_time_source("gpsdo");<br />
// This is equivalent to the previous commands, but faster, as it sets<br />
// both settings simultaneously and avoids duplicating settings that are shared<br />
// between these calls.<br />
usrp->set_sync_source("clock_source=gpsdo,time_source=gpsdo");<br />
</pre><br />
<br />
Note the GPS module is not always enabled. Its power-on status can be queried using the gps_enabled GPS sensor (see also The Sensor API). When disabled, none of the sensors will return useful (if any) values.<br />
<br />
When selecting gpsdo as a clock source, the GPS will always be enabled. Note that acquiring a GPS lock can take some time after enabling the GPS, so if a UHD application is enabling the GPS dynamically, it might take some time before a GPS lock is reported.<br />
<br />
==Front-Panel Programmable GPIOs==<br />
<br />
The USRP X4x0 has two HDMI front-panel connectors, which are connected to the FPGA. For a <br />
description of the GPIO control API, see the<br />
[https://files.ettus.com/manual/page_x400_gpio_api.html USRP X4x0 GPIO UHD Manual Entry],<br />
[https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_gpio the USRP X4x0 Series Manual],<br />
the [https://files.ettus.com/manual/page_zbx.html#zbx_atr ZBX ATR section] (X410) and the<br />
[https://files.ettus.com/manual/page_fbx.html#fbx_atr FBX ATR section] (X440).<br />
<br />
<br />
==Subdev Specifications==<br />
<br />
The RF ports on the front panel of the X410 + ZBX correspond to the following subdev specifications:<br />
<br />
{| class="wikitable" <br />
|-<br />
! Label<br />
! style="text-align:center; vertical-align:middle; font-weight:bold;" | Subdev Spec<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 0<br />
| A:0<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 1<br />
| A:1<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 0<br />
| B:0<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 1<br />
| B:1<br />
|}<br />
<br />
The RF ports on the front panel of the X440 + FBX correspond to the following subdev specifications (for xx_400 FPGA images):<br />
<br />
{| class="wikitable" <br />
|-<br />
! Label<br />
! style="text-align:center; vertical-align:middle; font-weight:bold;" | Subdev Spec<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 0<br />
| A:0<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 1<br />
| A:1<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 2<br />
| A:2<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 3<br />
| A:3<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 0<br />
| B:0<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 1<br />
| B:1<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 2<br />
| B:2<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 3<br />
| B:3<br />
|}<br />
<br />
When using a xx_1600 FPGA image on X440, only A:0 and B:0 are available.<br />
<br />
The subdev spec slot identifiers "A" and "B" are not reflected on the front panel. They were set to match valid subdev specifications of previous USRPs, maintaining backward compatibility.<br />
<br />
These values can be used for uhd::usrp::multi_usrp::set_rx_subdev_spec() and uhd::usrp::multi_usrp::set_tx_subdev_spec() as with other USRPs.<br />
<br />
<br />
==Rear Panel Status LEDs==<br />
<br />
The USRP X4x0 is equipped with four LEDs located on the device's rear panel. Each LED supports four different states: Off, Green, Red, and Amber. One LED (PWR) indicates the device's power state (see Power LED below). The other three LEDs (LED 0, LED 1, and LED 2) are user-configurable, different behaviors are supported for each of these LEDs (see User-configurable LEDs below).<br />
<br />
[[File:x4xx_rearpanel_status_leds.png|125px]]<br />
<br />
===X4x0 Rear Panel Status LEDs===<br />
Power LED<br />
The USRP X4x0's PWR LED is reserved to visually indicate the user the device's power state. Power LED Behavior describes what each LED state represents.<br />
<br />
===Power LED Behavior===<br />
<br />
{| class="wikitable" style="background-color:#FFF;"<br />
|- style="font-weight:bold; text-align:center;"<br />
! PWR LED State<br />
! style="vertical-align:middle;" | Meaning<br />
|- style="vertical-align:middle;"<br />
| Off<br />
| No power is applied<br />
|- style="vertical-align:middle;"<br />
| Amber<br />
| Power is good but X4x0 is powered off<br />
|- style="vertical-align:middle;"<br />
| Green<br />
| Power is good and X4x0 is powered on<br />
|- style="vertical-align:middle;"<br />
| Red<br />
| Power error state<br />
|}<br />
<br />
===User-configurable LEDs===<br />
The USRP X4x0's user-configurable rear panel status LEDs (LED 0, LED 1, and LED 2) allow the user to have visual indication of various device conditions. Supported LED Behaviors provides a complete list of the supported behaviors for each user-configurable LED. By default, these LEDs are configured as described in LEDs Default Behavior.<br />
<br />
The user may alter the default LEDs behavior either temporarily or persistently, see the Temporarily change the LED Behavior or Persistently in the UHD manual to change the LED Behavior accordingly.<br />
<br />
https://files.ettus.com/manual/page_usrp_x4xx.html<br />
<br />
==Technical Support and Community Knowledge Base==<br />
Technical support for USRP hardware is available through email only. If the product arrived in a nonfunctional state or you require technical assistance, please contact [mailto:support@ettus.com support@ettus.com]. Please allow 24 to 48 hours for response by email, depending on holidays and weekends, although we are often able to reply more quickly than that.<br />
<br />
We also recommend that you subscribe to the community mailing lists. The mailing lists have a responsive and knowledgeable community of hundreds of developers and technical users who are located around the world. When you join the community, you will be connected to this group of people who can help you learn about SDR and respond to your technical and specific questions. Often your question can be answered quickly on the mailing lists. Each mailing list also provides an archive of all past conversations and discussions going back many years. Your question or problem may have already been addressed before, and a relevant or helpful solution may already exist in the archive.<br />
<br />
Discussions involving the USRP hardware and the UHD software itself are best addressed through the '''usrp-users''' mailing list at [http://usrp-users.ettus.com http://usrp-users.ettus.com].<br />
<br />
Discussions involving the use of [http://gnuradio.org/ GNU Radio] with USRP hardware and UHD software are best addressed through the '''discuss-gnuradio''' mailing list at [https://lists.gnu.org/mailman/listinfo/discussgnuradio https://lists.gnu.org/mailman/listinfo/discussgnuradio].<br />
<br />
Discussions involving the use of [http://openbts.org/ OpenBTS®] with USRP hardware and UHD software are best addressed through the '''openbts-discuss''' mailing list at [https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss].<br />
<br />
The support page on our website is located at [https://www.ettus.com/support https://www.ettus.com/support]. The Knowledge Base is located at [https://kb.ettus.com https://kb.ettus.com].<br />
<br />
==Legal Considerations==<br />
Every country has laws governing the transmission and reception of radio signals. Users are solely responsible for insuring they use their USRP system in compliance with all applicable laws and regulations. Before attempting to transmit and/or receive on any frequency, we recommend that you determine what licenses may be required and what restrictions may apply.<br />
<br />
*NOTE: This USRP product is a piece of test equipment.<br />
<br />
==Sales and Ordering Support==<br />
If you have any non-technical questions related to your order, then please contact us by email at [mailto:orders@ettus.com orders@ettus.com], or by phone at +14086106399 (Monday-Friday, 8 AM - 5 PM, Pacific Time). Please be sure to include your order number and the serial number of your USRP.<br />
<br />
==Terms and Conditions of Sale==<br />
Terms and conditions of sale can be accessed online at the following link: http://www.ettus.com/legal/terms-and-conditions-of-sale<br />
<br />
[[Category:Getting Started Guides]]<br />
[[Category:X4x0]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6003Getting Started with DPDK and UHD2024-01-10T16:31:34Z<p>MichaelDickens: /* Stopping Extraneous Processes */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
systemctl stop containerd.socket<br />
systemctl stop docker.socket<br />
systemctl stop snapd.socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6002Getting Started with DPDK and UHD2024-01-10T16:29:57Z<p>MichaelDickens: /* Known Issues / Troubleshooting */ add multi-CPU systems starting point</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
systemctl stop containerd-socket<br />
systemctl stop docker-socket<br />
systemctl stop snapd-socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Regular Overflows on multi-CPU Systems ===<br />
<br />
On Multi-CPU systems each CPU is a NUMA node. The NUMA node contains all of the cores on the CPU. For best performance when using CPU/core isolation ("isocpus" per AAA above) make sure that all of the cores are in the same NUMA node.<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6001Getting Started with DPDK and UHD2024-01-09T20:57:15Z<p>MichaelDickens: /* Limit Execution CPUs/Cores */ move from 1-5 to 2-4</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 2 through and including 4, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "2-4"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
systemctl stop containerd-socket<br />
systemctl stop docker-socket<br />
systemctl stop snapd-socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=6000Getting Started with DPDK and UHD2024-01-09T20:56:30Z<p>MichaelDickens: /* Stopping Extraneous Processes */ add "socket" options</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "1-5"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code> and some might not exist on your specific system and for these the command will do nothing so it's OK to run it:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
systemctl stop containerd-socket<br />
systemctl stop docker-socket<br />
systemctl stop snapd-socket<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5918Getting Started with DPDK and UHD2023-11-21T19:30:03Z<p>MichaelDickens: /* UHD 4.x */ fix dpdk_driver for system install, not build from source default</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "1-5"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code>:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=USRP_Host_Performance_Tuning_Tips_and_Tricks&diff=5917USRP Host Performance Tuning Tips and Tricks2023-11-21T19:20:52Z<p>MichaelDickens: /* Disable Hyper-threading */ update with modern language</p>
<hr />
<div>==Application Note Number==<br />
'''AN-088'''<br />
<br />
==Overview==<br />
This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices.<br />
<br />
==CPU Governor==<br />
Ensure your CPU governor is set to <code>performance</code>. This can be done with the Linux utility <code>cpufrequtils</code>.<br />
<br />
Install <code>cpufrequtils</code> with the command below:<br />
<br />
sudo apt install cpufrequtils<br />
<br />
You can then set the CPU governor to <code>performance</code> per core by issuing the command:<br />
<br />
sudo cpufreq-set -c $core_number -g performance<br />
<br />
To set the CPU governor to <code>performance</code> for all cores:<br />
<br />
for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done<br />
<br />
You can then verify that the CPU governor has been set by running the command:<br />
<br />
cpufreq-info<br />
<br />
==Thread Priority Scheduling==<br />
<br />
When UHD spawns a new thread, it may try to boost the thread's scheduling priority. If setting the new priority fails, the UHD software prints a warning to the console, as shown below. This warning is harmless; it simply means that the thread will retain a normal or default scheduling priority.<br />
<br />
<pre><br />
UHD Warning:<br />
Unable to set the thread priority. Performance may be negatively affected.<br />
Please see the general application notes in the manual for instructions.<br />
EnvironmentError: OSError: error in pthread_setschedparam<br />
</pre><br />
<br />
To address this issue, non-privileged (non-root) users need to be given special permission to change the scheduling priority. This can be enabled by creating a group <code>usrp</code>, adding your user to it, and then appending the line <code>@usrp - rtprio 99</code> to the file <code>/etc/security/limits.conf</code>.<br />
<br />
sudo groupadd usrp<br />
sudo usermod -aG usrp $USER<br />
<br />
Then add the line below to end of the file <code>/etc/security/limits.conf</code>:<br />
<br />
@usrp - rtprio 99<br />
<br />
You must log out and log back into the account for the settings to take effect. In most Linux distributions, a list of groups and group members can be found in the <code>/etc/group</code> file.<br />
<br />
There is further documentation about this in the User Manual at the link below.<br />
<br />
* [http://files.ettus.com/manual/page_general.html#general_threading_prio Threading Notes section of the User Manual]<br />
<br />
<br />
==Adjust Network Buffers==<br />
This applies to USRP devices connected via Ethernet, such as the N200, N210, N300, N310, N320, N321, X300, X310, E320.<br />
<br />
Note that these settings will not persist across a reboot.<br />
<br />
sudo sysctl -w net.core.wmem_max=33554432<br />
sudo sysctl -w net.core.rmem_max=33554432<br />
sudo sysctl -w net.core.wmem_default=33554432<br />
sudo sysctl -w net.core.rmem_default=33554432<br />
<br />
==Adjust Ethernet MTU==<br />
This applies to Ethernet connected USRPs (N2xx, N3xx, X3xx, E320).<br />
<br />
For 1 Gigabit connections, the MTU should be set to <code>1500</code>.<br />
<br />
For 10 Gigabit connections, the MTU should be set to <code>9000</code>.<br />
<br />
It is important to set the value and '''not''' leave it is <code>automatic</code><br />
<br />
==Increasing Ring Buffers==<br />
This applies to Ethernet connected USRPs using a 10 Gb interface (X3xx, N3xx, E320).<br />
<br />
Increasing the Ring Buffers on the NIC may help prevent flow control errors at higher rates.<br />
<br />
sudo ethtool -G <interface> tx 4096 rx 4096<br />
<br />
== DPDK == <br />
DPDK is supported on N3xx, X3xx and E320 USRPs. DPDK replaces the traditional Linux networking stack with a low overhead user-land based driver. Additional details of using DPDK can be found in the UHD Manual located at the following link: https://files.ettus.com/manual/page_dpdk.html <br />
<br />
== Disable Hyper-threading ==<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
== Disable KPTI Protections for Spectre/Meltdown==<br />
In some cases, disabling the KPTI protections for the Linux Kernel can increase performance by 10-15%. It is important to note the ramification making this modification can have. This modification is only recommended for systems that absolutely require the best performance and are not connected to the internet.<br />
<br />
* https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)<br />
* https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)<br />
<br />
Disabling KPTI protections can be done by adding the lines below to your <code>/etc/default/grub</code> file at <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code><br />
<br />
pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier<br />
<br />
After modifying the <code>grub</code> file, run the following command to update your configuration and reboot:<br />
<br />
sudo update-grub<br />
<br />
<br />
<br />
<br />
<br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5916Getting Started with DPDK and UHD2023-11-21T19:20:24Z<p>MichaelDickens: redo clauses indenting</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
=== USRPs ===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs and lots of other NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD ==<br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running <code>cmake</code>.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code> and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You must note the MAC addresses for your NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You must then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
== Running UHD Applications with DPDK ==<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
== Tuning Notes ==<br />
<br />
=== General Host Performance Tuning App Note ===<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
=== Increasing <code>num_recv_frames</code> ===<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
=== Full Rate Streaming (UHD 3.x only) ===<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
=== Full Rate on X3xx ===<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
=== Isolate Cores/CPUs ===<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Disable System Interrupts on Cores/CPUs ===<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
=== Streaming on Multiple Channels using 1 Thread Per Stream ===<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
=== Elevated Streaming Thread Priority ===<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
=== Extra <code>nice</code> Priority ===<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
=== Limit Execution CPUs/Cores ===<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "1-5"<br />
<br />
=== Stopping Extraneous Processes ===<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code>:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
=== Disable Hyper-threading ===<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
=== Additional Tuning Notes from Intel ===<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5915Getting Started with DPDK and UHD2023-11-21T19:12:55Z<p>MichaelDickens: /* Installing DPDK */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
We recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
==== Increasing <code>num_recv_frames</code> ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
==== Isolate Cores/CPUs ====<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Disable System Interrupts on Cores/CPUs ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Streaming on Multiple Channels using 1 Thread Per Stream ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
==== Extra <code>nice</code> Priority ====<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
==== Limit Execution Cores ====<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "1-5"<br />
<br />
==== Stopping Extraneous Processes ====<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code>:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5914Getting Started with DPDK and UHD2023-11-21T18:57:09Z<p>MichaelDickens: /* Isolate Cores/CPUs */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Linux we recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
==== Increasing <code>num_recv_frames</code> ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
==== Isolate Cores/CPUs ====<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> (the "<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Disable System Interrupts on Cores/CPUs ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Streaming on Multiple Channels using 1 Thread Per Stream ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
==== Extra <code>nice</code> Priority ====<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
==== Limit Execution Cores ====<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "1-5"<br />
<br />
==== Stopping Extraneous Processes ====<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code>:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5913Getting Started with DPDK and UHD2023-11-21T18:50:36Z<p>MichaelDickens: /* Tuning Notes */ add Stopping Extraneous Processes</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Linux we recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
==== Increasing <code>num_recv_frames</code> ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
==== Isolate Cores/CPUs ====<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> ("<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Disable System Interrupts on Cores/CPUs ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Streaming on Multiple Channels using 1 Thread Per Stream ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
==== Extra <code>nice</code> Priority ====<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
==== Limit Execution Cores ====<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "1-5"<br />
<br />
==== Stopping Extraneous Processes ====<br />
<br />
The Linux kernel spawns a number of processes and threads that spend most of their time sleeping; one cannot stop these processes/threads. That said, in a typical Linux install (for example Ubuntu) there will be a graphical desktop (e.g., <code>gdm</code>) and various daemons started up by user request (e.g., <code>containerd</code>, <code>docker</code>, <code>snapd</code>). Most of these daemons can be controlled by <code>systemctl</code>; some must be manually quit. Any process that runs regularly stands a chance of interrupting the UHD and DPDK threads, and thus stopping, quitting, or disabling those processes can increase performance. For example, the following commands stop various daemons that execute regularly on Ubuntu; these need to be executed with <code>sudo</code>:<br />
<br />
systemctl stop containerd<br />
systemctl stop docker<br />
systemctl stop snapd<br />
<br />
The following command stops the main Ubuntu desktop GUI, so running it will render the system accessible only via networking (e.g., <code>ssh</code>), so make sure network access is enabled before executing this command:<br />
<br />
systemctl stop gdm<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5912Getting Started with DPDK and UHD2023-11-21T18:38:44Z<p>MichaelDickens: /* Limit Execution Cores */ no sudo required</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Linux we recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
==== Increasing <code>num_recv_frames</code> ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
==== Isolate Cores/CPUs ====<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> ("<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Disable System Interrupts on Cores/CPUs ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Streaming on Multiple Channels using 1 Thread Per Stream ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
==== Extra <code>nice</code> Priority ====<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
==== Limit Execution Cores ====<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following (note that <code>sudo</code> is ''not'' required):<br />
<br />
taskset -c "1-5"<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5911Getting Started with DPDK and UHD2023-11-21T18:37:31Z<p>MichaelDickens: /* Underruns Every Second with DPDK + Ubuntu */ update RT_RUNTIME_SHARE to be more precise and concise</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Linux we recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
==== Increasing <code>num_recv_frames</code> ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
==== Isolate Cores/CPUs ====<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> ("<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Disable System Interrupts on Cores/CPUs ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Streaming on Multiple Channels using 1 Thread Per Stream ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
==== Extra <code>nice</code> Priority ====<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
==== Limit Execution Cores ====<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following:<br />
<br />
sudo taskset -c "1-5"<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
NO_RT_RUNTIME_SHARE<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features | tr ' ' '\n' | grep RUNTIME_SHARE<br />
RT_RUNTIME_SHARE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5910Getting Started with DPDK and UHD2023-11-21T18:29:15Z<p>MichaelDickens: /* Tuning Notes */ add nice and taskset, some other tweaks, and rearrange ordering; add links to Linux commands</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Linux we recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
Perform the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks general host performance tuning tips and tricks].<br />
<br />
==== Increasing <code>num_recv_frames</code> ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
skip_ddc=1<br />
skip_duc=1<br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the following device arg:<br />
<br />
enable_tx_dual_eth=1<br />
<br />
==== Isolate Cores/CPUs ====<br />
<br />
Isolating the cores that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>/etc/default/grub</code> file in the <code>GRUB_CMDLINE_LINUX_DEFAULT=""</code> ("<code>GRUB_CONFIG</code>"). For example, to isolate cores 2 through and including 4, add this entry:<br />
<br />
isolcpus=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Disable System Interrupts on Cores/CPUs ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters <code>nohz_full</code> and <code>rcu_nocbs</code> to your <code>GRUB_CONFIG</code>. For example, to disable system interrupts on cores 2 through and including 4, add this entry:<br />
<br />
nohz_full=2-4 rcu_nocbs=2-4<br />
<br />
NOTE: After saving the <code>GRUB_CONFIG</code> file, execute <code>sudo update-grub</code> and then reboot the computer for the changes to take effect.<br />
<br />
==== Streaming on Multiple Channels using 1 Thread Per Stream ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the <code>benchmark_rate</code> example by using the parameter <code>--multi_streamer</code>.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the <code>uhd::set_thread_priority_safe()</code> function call. This can be accomplished with the <code>benchmark_rate</code> example by using parameter <code>--priority high</code>. Note that if the [https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Thread_Priority_Scheduling Thread Schedule Priority] has not been enabled for the current user then this parameter requires <code>sudo</code> to work.<br />
<br />
==== Extra <code>nice</code> Priority ====<br />
<br />
Beyond elevating streaming thread priority, one can also increase the [https://www.man7.org/linux/man-pages/man1/nice.1.html <code>nice</code> priority level] to maximum to increase the amount of CPU time for the process and its thread by prepending the following to the command being issued:<br />
<br />
sudo nice -n -20<br />
<br />
==== Limit Execution Cores ====<br />
<br />
With Linux one can limit the cores being used by the threads in a process via a [https://man7.org/linux/man-pages/man1/taskset.1.html <code>taskset</code>] prepended to a command being executed. When combining with the other techniques listed here, one can fairly well constrain a process and its thread to specific cores and give them maximum CPU time. Note that when using DPDK the MAC cores specified in <code>uhd.conf</code> will already be included as part of the <code>taskset</code> (but those cores will not be isolated nor have system interrupts disabled on them unless using that setting as noted above); it generally won't hurt to include the DPDK cores as part of the overall <code>taskset</code> but it's not required. For example, to limit process/thread execution to cores 1 through and including 5, one would prepend the following:<br />
<br />
sudo taskset -c "1-5"<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having fewer core threads. Hyper-threading is disabled within the BIOS and how to do this varies by motherboard manufacturer. With other techniques listed here, disabling hyper-threading should only be done as a last resort to eek absolute maximum performance from the CPU.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5909Getting Started with DPDK and UHD2023-11-21T15:40:20Z<p>MichaelDickens: /* UHD 4.0 */ various</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Linux we recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.x are slightly different from UHD 3.x.<br />
<br />
You must verify and/or update the following fields for your configuration from this example:<br />
<br />
* The MAC address variables, <code>dpdk_mac</code>, to match your NIC(s) link(s)<br />
<br />
* The <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on when DPDK is built and installed from source.<br />
<br />
* The <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC and both links are used. There must be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code> because it is not otherwise used in the file) and then separate cores assigned to each NIC link (in this example <code>core #3</code> for the one of the NIC links and <code>core #4</code> for the other NIC link).<br />
<br />
* The <code>dpdk_ipv4</code> fields to your desired IP range(s).<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
** <code>192.168.10.2</code> and <code>192.168.20.2</code> on a default X4xx systems<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Notes:'''<br />
* Additional information on the [https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config UHD DPDK configuration file is in the UHD manual].<br />
* The number of cores listed must be used exactly within the file.<br />
* The number of NIC link(s) listed must exactly match those specified in the UHD instantiation args.<br />
* A simple way to move between different DPDK configurations is to create different files and then symlink from the desired file to <code>uhd.conf</code><br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5908Getting Started with DPDK and UHD2023-11-21T15:26:34Z<p>MichaelDickens: /* Installing DPDK */ more tweaks</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Linux we recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-17.11/linux_gsg/build_dpdk.html DPDK 17.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
DPDK releases come twice per year in April (version X.04) and November (version X.11). The first release is more of a beta while the second is more of a formal release. While either ''can'' be used with UHD, we recommend just the formal releases with version ending in 11.<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.0 ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.0 are slightly different from UHD 3.x.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk_mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on Ubuntu 18.04.x when DPDK 18.11 is manually built and installed.<br />
<br />
* Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk_ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5907Getting Started with DPDK and UHD2023-11-21T15:23:37Z<p>MichaelDickens: /* Installing DPDK */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Linux we recommend installing DPDK via the system-provided installer; for example with Ubuntu:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to ''correctly'' build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.0 ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.0 are slightly different from UHD 3.x.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk_mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on Ubuntu 18.04.x when DPDK 18.11 is manually built and installed.<br />
<br />
* Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk_ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5906Getting Started with DPDK and UHD2023-11-21T15:21:21Z<p>MichaelDickens: /* Dependencies */ update UHD & DPDK verion options</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11<br />
<br />
* UHD 4.0 and 4.1 require DPDK 18.11<br />
<br />
* UHD 4.2 through 4.6 can use any version of DPDK from 18.11 through and including 21.11<br />
<br />
== Installing DPDK == <br />
<br />
On Ubuntu it is recommended to install DPDK via <code>apt</code>:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to correctly build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.0 ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.0 are slightly different from UHD 3.x.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk_mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on Ubuntu 18.04.x when DPDK 18.11 is manually built and installed.<br />
<br />
* Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk_ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5905Getting Started with DPDK and UHD2023-11-21T15:14:13Z<p>MichaelDickens: /* Installing DPDK */ various updates</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11, which is included in the default repos of Ubuntu 18.04.x<br />
<br />
* UHD 4.0 requires DPDK 18.11<br />
<br />
* DPDK support was added for the N3xx/E320 USRPs with UHD 3.13.x.x <br />
<br />
* DPDK support was added for the X3xx with UHD 3.14.1.0<br />
<br />
== Installing DPDK == <br />
<br />
On Ubuntu it is recommended to install DPDK via <code>apt</code>:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
While it is possible to install DPDK from source, we recommend using the system-provided install unless there is a very good reason to not do so. DPDK can be challenging to correctly build from source. If you require installing DPDK from source, the install guide for various versions is noted below:<br />
<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 18.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 19.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 20.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 21.11]<br />
* [https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html DPDK 22.11]<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.0 ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.0 are slightly different from UHD 3.x.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk_mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on Ubuntu 18.04.x when DPDK 18.11 is manually built and installed.<br />
<br />
* Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk_ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5904Getting Started with DPDK and UHD2023-11-21T15:10:29Z<p>MichaelDickens: /* References */ add DPDK versions used to current</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
** https://doc.dpdk.org/guides-19.11<br />
** https://doc.dpdk.org/guides-20.11<br />
** https://doc.dpdk.org/guides-21.11<br />
** https://doc.dpdk.org/guides-22.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11, which is included in the default repos of Ubuntu 18.04.x<br />
<br />
* UHD 4.0 requires DPDK 18.11<br />
<br />
* DPDK support was added for the N3xx/E320 USRPs with UHD 3.13.x.x <br />
<br />
* DPDK support was added for the X3xx with UHD 3.14.1.0<br />
<br />
== Installing DPDK == <br />
<br />
On Ubuntu 18.04.x, it is possible to install DPDK 17.11 via <code>apt</code>:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
For DPDK 18.11, follow the instructions on the DPDK website to download, configure, and build DPDK (https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html).<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.0 ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.0 are slightly different from UHD 3.x.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk_mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on Ubuntu 18.04.x when DPDK 18.11 is manually built and installed.<br />
<br />
* Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk_ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5903Getting Started with DPDK and UHD2023-11-20T18:01:50Z<p>MichaelDickens: /* Host Network Cards */ update list</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb and 100Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1 (1x10Gb)<br />
* Intel X520-DA2 (2x10Gb)<br />
* Intel X710-DA2 (2x10Gb)<br />
* Intel X710-DA4 (4x10Gb)<br />
* Intel XL710-QDA2 (2x40Gb breakout to 4x10Gb)<br />
* Intel E810-CQDA1 & E810-CQDA2 (2x100Gb and 2x4x10Gb)<br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx (2x10Gb)<br />
* Mellanox MCX515A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CCAT ConnectX-5 EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX516A-CDAT ConnectX-5 Ex EN (2x100Gb or 2x10Gb)<br />
* Mellanox MCX623106AN-CDAT ConnectX-6 Dx EN (2x100Gb or 2x10Gb)<br />
* [https://www.ni.com/en-us/support/model.100-gb-ethernet-connectivity-kit.html NI Dual 100 Gigabit Ethernet PCIe Interface Kit (PN 788216-01)]<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11, which is included in the default repos of Ubuntu 18.04.x<br />
<br />
* UHD 4.0 requires DPDK 18.11<br />
<br />
* DPDK support was added for the N3xx/E320 USRPs with UHD 3.13.x.x <br />
<br />
* DPDK support was added for the X3xx with UHD 3.14.1.0<br />
<br />
== Installing DPDK == <br />
<br />
On Ubuntu 18.04.x, it is possible to install DPDK 17.11 via <code>apt</code>:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
For DPDK 18.11, follow the instructions on the DPDK website to download, configure, and build DPDK (https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html).<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.0 ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.0 are slightly different from UHD 3.x.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk_mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on Ubuntu 18.04.x when DPDK 18.11 is manually built and installed.<br />
<br />
* Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk_ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5902Getting Started with DPDK and UHD2023-11-20T17:24:43Z<p>MichaelDickens: /* USRPs */ reorder & add X410 * X440</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* E320<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310<br />
* X410<br />
* X440<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1<br />
* Intel X520-DA2<br />
* Intel X710-DA2<br />
* Intel X710-DA4<br />
* Intel XL710 <br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx <br />
* Mellanox MCX516A-CCAT ConnectX-5<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11, which is included in the default repos of Ubuntu 18.04.x<br />
<br />
* UHD 4.0 requires DPDK 18.11<br />
<br />
* DPDK support was added for the N3xx/E320 USRPs with UHD 3.13.x.x <br />
<br />
* DPDK support was added for the X3xx with UHD 3.14.1.0<br />
<br />
== Installing DPDK == <br />
<br />
On Ubuntu 18.04.x, it is possible to install DPDK 17.11 via <code>apt</code>:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
For DPDK 18.11, follow the instructions on the DPDK website to download, configure, and build DPDK (https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html).<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.0 ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.0 are slightly different from UHD 3.x.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk_mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on Ubuntu 18.04.x when DPDK 18.11 is manually built and installed.<br />
<br />
* Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk_ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Getting_Started_with_DPDK_and_UHD&diff=5901Getting Started with DPDK and UHD2023-11-20T17:23:48Z<p>MichaelDickens: /* Application Note Number and Authors */ add myself, getting ready for "lots of updates"</p>
<hr />
<div>== Application Note Number and Authors ==<br />
<br />
'''AN-500''' by Nate Temple, Alex Williams, Wade Fife, Matt Prost, and Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-01-08<br />
|style="text-align:center;"| Nate Temple & Alex Williams<br />
|style="text-align:center;"| Initial creation<br />
|-<br />
|style="text-align:center;"| 2021-02-25<br />
|style="text-align:center;"| Wade Fife<br />
|style="text-align:center;"| Added corrections specific to UHD 4.0 and Mellanox card use.<br />
|-<br />
|style="text-align:center;"| 2022-02-08<br />
|style="text-align:center;"| Matt Prost (via Michael Dickens)<br />
|style="text-align:center;"| Tweak Tuning Notes and add Known Issues<br />
|-<br />
|style="text-align:center;"| 2023-11-20<br />
|style="text-align:center;"| Michael Dickens<br />
|style="text-align:center;"| Lots of Updates<br />
|}<br />
--><br />
<br />
==Overview==<br />
<br />
This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
<br />
==Abstract==<br />
<br />
Up until now, UHD's only support for networked devices was backed by the kernel's sockets implementation. Every call to <code>send()</code> or <code>recv()</code> would cause a context switch and invite the kernel's scheduler to replace our thread with something else. Because the typical scheduler is optimized to distribute CPU time fairly across multiple loads, the timing-critical threads might sporadically be hit with sleeping time, and the thread might be migrated off its current CPU and forced to run on another. The overhead and random latency spikes make it difficult to enable reliable real-time streaming at higher rates.<br />
<br />
DPDK is a high-speed packet processing framework that enables a kernel bypass for network drivers. By putting the entire driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the scheduler. In addition, the overall overhead for packet processing lowers. <br />
<br />
== Supported Devices ==<br />
<br />
===USRPs===<br />
<br />
DPDK is supported on the following USRP devices:<br />
<br />
* N300 / N310<br />
* N320 / N321<br />
* X300 / X310 <br />
* E320<br />
<br />
=== Host Network Cards ===<br />
<br />
DPDK is supported on many Intel and Mellanox based 10Gb NICs. Below is a list of NICs Ettus Research has tested. For a full list of NICs supported by DPDK, please see the DPDK manual.<br />
<br />
* Intel X520-DA1<br />
* Intel X520-DA2<br />
* Intel X710-DA2<br />
* Intel X710-DA4<br />
* Intel XL710 <br />
* Mellanox MCX4121A-ACAT ConnectX-4 Lx <br />
* Mellanox MCX516A-CCAT ConnectX-5<br />
<br />
== References ==<br />
<br />
* DPDK: https://www.dpdk.org/<br />
** https://doc.dpdk.org/guides-17.11<br />
** https://doc.dpdk.org/guides-18.11<br />
* UHD Manual: https://files.ettus.com/manual/page_dpdk.html<br />
<br />
== Dependencies ==<br />
<br />
* UHD 3.x requires DPDK 17.11, which is included in the default repos of Ubuntu 18.04.x<br />
<br />
* UHD 4.0 requires DPDK 18.11<br />
<br />
* DPDK support was added for the N3xx/E320 USRPs with UHD 3.13.x.x <br />
<br />
* DPDK support was added for the X3xx with UHD 3.14.1.0<br />
<br />
== Installing DPDK == <br />
<br />
On Ubuntu 18.04.x, it is possible to install DPDK 17.11 via <code>apt</code>:<br />
<br />
sudo apt install dpdk dpdk-dev<br />
<br />
For DPDK 18.11, follow the instructions on the DPDK website to download, configure, and build DPDK (https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html).<br />
<br />
== Installing UHD == <br />
<br />
Once the <code>dpdk</code> and <code>dpdk-dev</code> packages are installed, UHD will locate them during a build and you should see DPDK in the enabled components lists when running cmake.<br />
<br />
== Enable hugepages ==<br />
<br />
Edit your grub configuration file, <code>/etc/default/grub</code>, and add the follow parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
<br />
iommu=pt intel_iommu=on hugepages=2048<br />
<br />
On a vanilla Ubuntu system it should look like this:<br />
<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048"<br />
<br />
Close <code>/etc/default/grub</code> and at the command prompt, update your grub configuration with the command:<br />
<br />
sudo update-grub<br />
<br />
For these settings to take effect, reboot your host machine.<br />
<br />
== Preparing your UHD Configuration File ==<br />
<br />
You should note the MAC addresses for your 10Gb NICs before proceeding.<br />
<br />
The MAC addresses for your NICs can be found by running the command:<br />
<br />
ip a<br />
<br />
You should then create a UHD configuration file at the location <code>/root/.uhd/uhd.conf</code>.<br />
<br />
sudo su<br />
mkdir -p /root/.uhd<br />
nano /root/.uhd/uhd.conf<br />
<br />
=== UHD 3.x ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that field names in UHD 3.x are slightly different from UHD 4.0.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk-mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk-driver</code> if the location is different on your system. <code>/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/</code> is the default location on Ubuntu 18.04.x when <code>dpdk</code> is installed via <code>apt</code>.<br />
<br />
* Update the <code>dpdk-corelist</code> and <code>dpdk-io-cpu</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk-ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk-mtu=9000<br />
dpdk-driver=/usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/<br />
dpdk-corelist=2,3,4<br />
dpdk-num-mbufs=4095<br />
dpdk-mbufs-cache-size=315<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f1]<br />
dpdk-io-cpu = 3<br />
dpdk-ipv4 = 192.168.10.1/24<br />
<br />
[dpdk-mac=aa:bb:cc:dd:ee:f2]<br />
dpdk-io-cpu = 4<br />
dpdk-ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual_archive/v3.15.0.0/html/page_dpdk.html#dpdk_nic_config<br />
<br />
=== UHD 4.0 ===<br />
<br />
An example <code>uhd.conf</code> file is listed below. Note that the field names in UHD 4.0 are slightly different from UHD 3.x.<br />
<br />
You should update the following fields for your configuration from this example:<br />
<br />
* Update the MAC address variables, <code>dpdk_mac</code>, to match your NIC<br />
<br />
* Update the <code>dpdk_driver</code> if the location is different on your system. <code>/usr/local/lib/</code> is the default location on Ubuntu 18.04.x when DPDK 18.11 is manually built and installed.<br />
<br />
* Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two port NIC is used. There should be one core for the main <code>dpdk</code> thread (in this example <code>core #2</code>), and then separate cores assigned to each NIC (in this example <code>core #3</code> for the first port on the NIC, <code>core #4</code> for the second port on the NIC)<br />
<br />
* Update the <code>dpdk_ipv4</code> fields to your desired IP range.<br />
** <code>192.168.30.2</code>, <code>192.168.40.2</code> on a default X3xx system<br />
** <code>192.168.10.2</code>, <code>192.168.20.2</code> on a default N3xx system<br />
** <code>192.168.10.2</code> on a default E320 system<br />
<br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=4095<br />
dpdk_mbuf_cache_size=315<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f1]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.10.1/24<br />
<br />
[dpdk_mac=aa:bb:cc:dd:ee:f2]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.20.1/24<br />
<br />
'''Note:''' Additional information on the UHD configuration file can be found here: https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
== Additional Host Configuration for NIC Vendors ==<br />
<br />
The process for this step is different for Intel and Mellanox NICs and is detailed in individual sections below.<br />
<br />
=== Intel X520 / X710 ===<br />
<br />
The Intel based NICs will use the <code>vfio-pci</code> driver which must be loaded:<br />
<br />
sudo modprobe vfio-pci<br />
<br />
Next, you will need to rebind the NIC to the <code>vfio-pci</code> drivers. <br />
<br />
First, identify the PCI address your NIC is at:<br />
<br />
dpdk-devbind --status<br />
<br />
Note the PCI address that your NIC is connected to for the next step.<br />
<br />
Before the next step, you will need to turn off the NIC first before doing the rebind. <br />
<br />
In Ubuntu under <code>System</code> -> <code>Network</code> -> click the switches to <code>off</code> for the 10Gb ports, then run the <code>dpdk-devbind</code> commands:<br />
<br />
'''Note:''' Your PCI address will likely be different than <code>02:00.0</code> as shown in the example below.<br />
<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.0<br />
sudo dpdk-devbind --bind=vfio-pci 02:00.1<br />
<br />
Now if you run <code>dpdk-devbind --status</code> again, you should see the NICs listed under DPDK devices<br />
<br />
# dpdk-devbind --status<br />
<br />
Network devices using DPDK-compatible driver<br />
============================================<br />
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe<br />
<br />
<br />
'''Note:''' More info can be found here on the rebinding process: https://doc.dpdk.org/guides-17.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules<br />
<br />
=== Mellanox NICs ===<br />
<br />
The Mellanox NICs do not require rebinding using the <code>vfio-pci</code> driver. Mellanox provides additional drivers for DPDK.<br />
<br />
Install and activate the Mellanox drivers:<br />
<br />
sudo apt install librte-pmd-mlx5-17.11<br />
sudo modprobe -a ib_uverbs mlx5_core mlx5_ib<br />
<br />
For 18.11 you can download and install the latest Mellanox drivers from the mellanox website (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed).<br />
<br />
The MLX5 poll mode driver library (librte_pmd_mlx5) in DPDK provides support for Mellanox ConnextX-4 and ConnectX-5 cards. This driver must be enabled manually with the build option <code>CONFIG_RTE_LIBRTE_MLX5_PMD=y</code> when building DPDK.<br />
<br />
=== Running UHD Applications with DPDK ===<br />
<br />
UHD based application (including GNU Radio flowgraphs) can now be ran using a DPDK transport by passing in the Device Argument: <code>use_dpdk=1</code>.<br />
<br />
'''Important Note:''' In order for UHD to use DPDK, the UHD application *must* be ran as the <code>root</code> user. Using <code>sudo</code> will not work, you should switch to the <code>root</code> user by running <code>sudo su</code>.<br />
<br />
For example, running the <code>benchmark_rate</code> utility:<br />
<br />
<pre><br />
# cd /usr/local/lib/uhd/examples<br />
<br />
# ./benchmark_rate --rx_rate 125e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 125e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1"<br />
<br />
[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-gabf0db4e<br />
EAL: Detected 8 lcore(s)<br />
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA<br />
EAL: No free hugepages reported in hugepages-1048576kB<br />
EAL: Probing VFIO support...<br />
EAL: VFIO support initialized<br />
EAL: PCI device 0000:02:00.0 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: using IOMMU type 1 (Type 1)<br />
EAL: Ignore mapping IO port bar(2)<br />
EAL: PCI device 0000:02:00.1 on NUMA socket -1<br />
EAL: Invalid NUMA socket, default to 0<br />
EAL: probe driver: 8086:10fb net_ixgbe<br />
EAL: Ignore mapping IO port bar(2)<br />
PMD: ixgbe_dev_link_status_print(): Port 0: Link Down<br />
EAL: Port 0 MAC: aa bb cc dd ee f1<br />
EAL: Port 0 UP: 1<br />
PMD: ixgbe_dev_link_status_print(): Port 1: Link Down<br />
EAL: Port 1 MAC: aa bb cc dd ee f2<br />
EAL: Port 1 UP: 1<br />
EAL: Init DONE!<br />
EAL: Starting I/O threads!<br />
USER2: Thread 1 started<br />
[00:00:00.000003] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=10.2.1.19,master_clock_rate=125e6,use_dpdk=1...<br />
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=10.2.1.19,type=n3xx,product=n310,serial=313ABDA,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=125e6,use_dpdk=1<br />
[INFO] [MPM.PeriphManager] init() called with device args 'product=n310,time_source=internal,master_clock_rate=125e6,clock_source=internal,use_dpdk=1,second_addr=192.168.20.2,mgmt_addr=10.2.1.19'.<br />
[INFO] [0/DmaFIFO_0] Initializing block control (NOC ID: 0xF1F0D00000000004)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1344 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1341 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1348 MB/s)<br />
[INFO] [0/DmaFIFO_0] BIST passed (Throughput: 1347 MB/s)<br />
[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)<br />
[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)<br />
[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)<br />
Using Device: Single USRP:<br />
Device: N300-Series Device<br />
Mboard 0: ni-n3xx-313ABDA<br />
RX Channel: 0<br />
RX DSP: 0<br />
RX Dboard: A<br />
RX Subdev: Magnesium<br />
RX Channel: 1<br />
RX DSP: 0<br />
RX Dboard: B<br />
RX Subdev: Magnesium<br />
TX Channel: 0<br />
TX DSP: 0<br />
TX Dboard: A<br />
TX Subdev: Magnesium<br />
TX Channel: 1<br />
TX DSP: 0<br />
TX Dboard: B<br />
TX Subdev: Magnesium<br />
<br />
[00:00:03.728707] Setting device timestamp to 0...<br />
[INFO] [MULTI_USRP] 1) catch time transition at pps edge<br />
[INFO] [MULTI_USRP] 2) set times next pps (synchronously)<br />
[00:00:05.331920] Testing receive rate 125.000000 Msps on 2 channels<br />
[00:00:05.610789] Testing transmit rate 125.000000 Msps on 2 channels<br />
[00:00:15.878071] Benchmark complete.<br />
<br />
<br />
Benchmark rate summary:<br />
Num received samples: 2557247854<br />
Num dropped samples: 0<br />
Num overruns detected: 0<br />
Num transmitted samples: 2504266704<br />
Num sequence errors (Tx): 0<br />
Num sequence errors (Rx): 0<br />
Num underruns detected: 0<br />
Num late commands: 0<br />
Num timeouts (Tx): 0<br />
Num timeouts (Rx): 0<br />
<br />
<br />
Done!<br />
</pre><br />
<br />
=== Tuning Notes ===<br />
<br />
==== General Host Performance Tuning App Note ====<br />
<br />
The Application Note linked below covers general performance tuning tips that should be applied:<br />
<br />
* https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks<br />
<br />
==== Increasing num_recv_frames ====<br />
<br />
If you experience <code>Overflows</code> at higher data rates, adding the device argument <code>num_recv_frames=512</code> can help.<br />
<br />
==== Full Rate Streaming (UHD 3.x only) ====<br />
<br />
If you're streaming data at the full master clock rate, and there is no interpolation or decimation being performed on the FPGA, you can skip the DUC and DDC blocks within the FPGA with the following parameters:<br />
<br />
* <code>skip_ddc=1</code><br />
* <code>skip_duc=1</code><br />
<br />
==== Full Rate on X3xx ====<br />
<br />
If you're streaming two transmit channels at full rate (200e6) on the X3xx platform, you should additionally set the device arg:<br />
<br />
* <code>enable_tx_dual_eth=1</code><br />
<br />
==== Isolate CPUs ====<br />
<br />
Isolating the CPUs that are used for DPDK can improve performance. This can be done by adding the <code>isolcpus</code> parameter to your <code>GRUB_CONFIG</code><br />
<br />
isolcpus=2,3,4<br />
<br />
==== Disable System Interrupts ====<br />
<br />
Disabling system interrupts can improve the jitter and performance generally by 1-3%. This can be done by adding the parameters below to your <code>GRUB_CONFIG</code><br />
<br />
nohz_full=2,3,4 rcu_nocbs=2,3,4<br />
<br />
==== Disable Hyper-threading ====<br />
<br />
In some applications which require the highest possible CPU performance per core, disabling hyper-threading can provide roughly a 10% increase in core performance, at the cost of having less core threads. Hyper-threading can be disabled within the BIOs and varies by manufacturer.<br />
<br />
==== Streaming on Multiple Channels ====<br />
<br />
If you're streaming on multiple channels simultaneously, you can create multiple streamer objects on separate threads. This can be accomplished with the `benchmark_rate` example by using the parameter `--multi_streamer`.<br />
<br />
==== Elevated Streaming Thread Priority ====<br />
<br />
In UHD 4, streaming thread priorities can be elevated with the `uhd::set_thread_priority_safe()` function call. This can be accomplished with the benchmark_rate example by using parameter `--priority high`.<br />
<br />
==== Additional Tuning Notes from Intel ====<br />
<br />
* Performance report from Intel on DPDK 17.11: https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_performance_report.pdf<br />
* How to get best performance with NICs on Intel platforms: https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html<br />
<br />
== Known Issues / Troubleshooting ==<br />
<br />
=== Underruns Every Second with DPDK + Ubuntu ===<br />
<br />
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems that otherwise have no issues. These Linux kernel versions are the default for Ubuntu 20.04.3 LTS and later. The underrun issue is due to the <code>RT_RUNTIME_SHARE</code> feature being disabled by default in these versions of the Linux kernel (shown as <code>NO_RT_RUNTIME_SHARE</code>). The following procedure can be used to enable this feature. This process was tested on Linux kernel version 5.13; the procedure may be slightly different on other kernel versions. To determine the Linux kernel version of your system, in a terminal issue the command <code>uname -r</code>.<br />
<br />
<pre><br />
$ sudo -s<br />
$ cd /sys/kernel/debug/sched<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''NO_RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
<br />
$ echo RT_RUNTIME_SHARE > features<br />
$ cat features<br />
<br />
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI '''RT_RUNTIME_SHARE''' NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=USRP_X410_Getting_Started_Guide&diff=5882USRP X410 Getting Started Guide2023-09-25T13:46:00Z<p>MichaelDickens: delete contents & move to REDIRECT since all of this content is now on X410/X440, but we want old search results to still work</p>
<hr />
<div>#REDIRECT[[USRP X410/X440 Getting Started Guide]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=X440&diff=5881X4402023-09-22T20:28:38Z<p>MichaelDickens: /* Certificate / Letter of Volatility */</p>
<hr />
<div>== Notice ==<br />
'''When you receive a brand-new device, it is strongly recommended that you download the most recent filesystem image from the Ettus Research website and write it to the NI Ettus USRP X4x0. Instructions on downloading the latest filesystem image and writing it to the X4x0 are described in the [https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide USRP X4x0 Getting Started Guide].'''<br />
<br />
'''Note that if you are operating the device in Network Mode, then the versions of UHD running on the host computer and on the NI Ettus USRP X4x0 device must match.'''<br />
<br />
== Device Overview ==<br />
===X440===<br />
The NI Ettus USRP X440 is the widest bandwidth USRP software defined radio device. It differs architecturally from most USRPs, utilizing a direct sampling architecture that provides balun-coupled access to the ADCs and DACs on the onboard Xilinx Zynq RFSoC. This makes it well suited to use as an intermediate frequency transceiver, connecting to external front ends for applications like satellite communications (SATCOM) prototyping, SATCOM ground station deployment, and mmWave or sub-THz 6G research. USRP X440 features high channel density – 8 Tx and 8 Rx channels per device – with phase coherency across channels by sharing sample clocks. Therefore, it is ideal for applications like direction finding and radar research and prototyping. <br />
<br />
== Key Features==<br />
<br />
===X440===<br />
{|<br />
|style="vertical-align:top"|<br />
* High channel density<br />
* Reliable and fault-tolerant deployment<br />
* Stand-alone (embedded) or host-based (network streaming) operation<br />
* Fully integrated and assembled (the USRP X440 does not support swappable daughtercards)<br />
* 30 MHz to 4 GHz frequency range (tunable down to 1MHz)<br />
* Up to 1600 MHz of instantaneous bandwidth per channel<br />
* 8 RX, 8 TX in half-wide RU form factor<br />
* Xilinx Zynq-Ultrascale+ ZU28DR RFSoC<br />
* 12 bit ADC, 14 bit DAC<br />
* IQ Sample Clock rates up to 2000 MS/s<br />
* Onboard SD-FEC, DDC, DUC<br />
* Quad-core ARM Cortex-A53 up to 1.2 GHz CPU<br />
* Dual-core ARM Cortex-A5 MPCore up to 500 MHz<br />
* Two QSFP28 ports (10 Gigabit Ethernet, 100 Gigabit Ethernet)<br />
* RJ45 (1 GbE) [1]<br />
* 10 MHz Clock reference <br />
* PPS time reference<br />
* Trig In/Out Interface<br />
* Built-in GPSDO <br />
* Two FPGA Programmable GPIO Interfaces (HDMI)<br />
* 1 Type C USB host port <br />
* 1 Type C USB port (serial console, JTAG) <br />
* Watchdog timer<br />
* OpenEmbedded Linux<br />
* USRP Hardware Driver™ (UHD) open-source software API version 4.5.0 or later<br />
* RF Network on Chip (RFNoC™) FPGA development framework<br />
* Xilinx Vivado® 2021.1 Design Suite (license not included)<br />
* GNU Radio support maintained by Ettus Research™ through GR-UHD, an interface to UHD distributed by GNU Radio<br />
* [1] The RJ45 port is used for remote management of the device and does not support IQ streaming.<br />
<br />
|[[File:X440.jpg|500px|center]] <br />
<br />
<br />
|}<br />
<br />
==Hardware Specifications==<br />
===X440===<br />
* Current Hardware Revision: Module revision D and Motherboard revision F<br />
* Minimum version of UHD required: 4.5.0<br />
* USRP X440 is covered by Export Administration Regulations (EAR) NS2. Refer to [https://www.bis.doc.gov/ US Department of Commerce] for details and country chart. <br />
<br />
https://www.ni.com/docs/en-US/bundle/ettus-usrp-x440-specs/page/specs.html<br />
https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0<br />
<br />
===CAD/STP Models===<br />
<br />
If you want any CAD / STP models beyond those found here, please send an email to Ettus Support at [mailto:support@ettus.com support@ettus.com] noting your request and your use case for any such model. We will determine on a case-by-case basis whether we have any such requested model and, if so, whether to release it -- possibly requiring an NDA for any such release. Note that we do not have models on all USRPs and daughterboards, and requesting any model does not guarantee that either Ettus Research or NI will honor any such request.<br />
<br />
==FPGA==<br />
===FPGA User Modifications===<br />
The Verilog code for the FPGA in the NI Ettus USRP X4x0 is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Specifically, changing the I/O interface of the FPGA in any way, or modifying the pin and timing constraint files, could result in physical damage to other components on the motherboard, external to the FPGA, and doing this will void the warranty. Also, even if the PCIe interface is not being used, you cannot remove or reassign these pins in the constraint file. The constraint files should not be modified. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
<br />
==Interfaces and Connectivity==<br />
<br />
===Front Panel===<br />
<br />
====X440====<br />
<br />
[[File:x440_front_panel.jpg|500px|center]] <br />
<br />
===Rear Panel===<br />
<br />
[[File:x440_back_panel.jpg|500px|center]]<br />
<br />
===Ref Clock - 10 MHz===<br />
Using an external 10 MHz reference clock, a square wave will offer the best phase noise performance, but a sinusoid is acceptable.<br />
<br />
===PPS - Pulse Per Second===<br />
Using a PPS signal for timestamp synchronization requires a square wave signal (a typical PPS signal has a 20%-25% duty cycle) with a 5 Vpp amplitude. <br />
<br />
To test the PPS input, you can use the following tool from the UHD examples:<br />
<br />
* <code><args></code> are device address arguments (optional if only one USRP device is on your machine)<br />
<br />
cd <install-path>/lib/uhd/examples ./test_pps_input –args=<args><br />
<br />
===Front Panel GPIO===<br />
* https://files.ettus.com/manual/page_x400_gpio_api.html#x4x0gpio_fpanel<br />
<br />
====Power on state====<br />
<br />
* https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide#Autoboot<br />
* https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_rearpanelleds_power<br />
<br />
==Accessories==<br />
<br />
For datasheet, drawings, pricing, and purchasing please search for the Part Number listed below via https://www.ni.com/en-us/search.html<br />
<br />
* Dual 100 Gigabit Ethernet PCIe Interface Kit For Ettus USRP X4xx<br />
** Part Number: 788216-01<br />
<br />
* USRP X4xx Power Supply, 100-240VAC 50/60HZ 12VDC 24AMP<br />
** Part Number: 788204-01<br />
<br />
* QSFP28 To 4xSFP28 Breakout Cable, 1M<br />
** Part Number: 788214-01<br />
<br />
* QSFP28 Twinaxial Cable, 3M <br />
** Part Number: 788215-03<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 2 USRP X4xx Devices, Shoulder to Shoulder <br />
** Part Number: 788147-01<br />
<br />
* USRP X4xx Desktop Stack Accessory, Single USRP X4xx Device Fastened Buildup <br />
** Part Number: 788148-01<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 1 USRP X4xx Device, w/Surrogate Extension <br />
** Part Number: 788149-01<br />
<br />
* GPIO Communication Cable<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 1M <br />
*** Part Number: 152629-01<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 2M <br />
*** Part Number: 152629-02<br />
<br />
* SCB-19 Noise Rejecting, Shielded Aux I/O Connector <br />
** Part Number: 782444-01<br />
<br />
* Fan Replacement Cartridges<br />
** USRP X4xx Fan Cartridge Accessory, Exhaust<br />
*** Part Number: 788164-01<br />
** USRP X4xx Fan Cartridge Accessory, Intake<br />
*** Part Number: 788165-01<br />
<br />
==10 Gigabit Ethernet==<br />
'''Recommended 10 Gigabit Ethernet Cards'''<br />
* Dual 10 Gigabit Ethernet Interface for Ettus USRP <br />
** [https://search.ni.com/nisearch/app/main/p/bot/no/ap/global/lang/en/pg/1/q/788600-01/ ni.com part number 788600-01]<br />
* Intel X710-DA2<br />
** [http://ark.intel.com/products/83964/Intel-Ethernet-Converged-Network-Adapter-X710-DA2 Intel® Ethernet Converged Network Adapter X710-DA2 ]<br />
* Intel X710-DA4<br />
** [http://ark.intel.com/products/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4 Intel® Ethernet Converged Network Adapter X710-DA4 ]<br />
<br />
==100 Gigabit Ethernet==<br />
<br />
===X440===<br />
<br />
* Requires UHD 4.5 or later: https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0 <br />
* 100GbE Streaming only supports Linux Hosts <br />
<br />
'''Recommended 100 Gigabit Ethernet Cards'''<br />
* Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16))<br />
<br />
'''Recommended 100 Gigabit Ethernet Cables'''<br />
* Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N<br />
** Shorter length variants also recommended<br />
<br />
'''Recommended Host PC'''<br />
* At least 32 CPU Cores<br />
* At least 64 GB RAM<br />
* Ubuntu 20.04 (5.13.0-44-generic)<br />
<br />
''' Validated Hardware and Software Configuration Examples '''<br />
* Ubuntu 20.04 (5.13.0-44-generic kernel), DPDK 20.11, with AMD Ryzen Threadripper 3960X 24-Core Processor - 48 CPU - 3.6 GHz CPU freq - 64 GB RAM. Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16)). Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N cables.<br />
<br />
''' Data Throughput Rates '''<br />
<br />
Testing was completed with the following conditions<br />
* Hardware and Software Configurations listed above<br />
* CPU configured for performance mode: https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#CPU_Governor<br />
* DPDK Setup: https://files.ettus.com/manual/page_dpdk.html and https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#UHD_4.0<br />
* Enabling Tx pause Frames on X4x0 for the SFP port(s) utilized for streaming: https://files.ettus.com/manual/page_transport.html#transport_udp_linux<br />
** <code>ethtool -A sfp0 tx on</code><br />
** <code>ethtool -A sfp1 tx on</code><br />
* uhd.conf: See https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
Executing [https://github.com/EttusResearch/uhd/blob/UHD-4.5/host/examples/benchmark_rate.cpp benchmark_rate ] over multiple iterations as well as over an extended continuous time period (>12 Hours) without data loss resulted in the following maximum rates and channel counts<br />
* Usage of the <code>--priority</code> argument set to <code>high</code> in benchmark_rate which requires benchmark rate to be executed with root privileges via <code>sudo</code><br />
* Usage of single versus multiple threads in the benchmark_rate utility - controlled by using the <code>--multi_streamer</code> argument. Specifying this argument assigns one thread per channel being streamed.<br />
* Utilizing the CG_400 and CG_1600 bitfile.<br />
<br />
* CG_400 <br />
** Dual Port DPDK <br />
*** 6 Rx @ 500 MS/s <br />
*** 8 Rx @ 400 MS/s <br />
*** 6 Tx @ 500 MS/s <br />
*** 8 Tx @ 450 MS/s <br />
*** 4 Rx + 4 Tx @ 500 MS/s <br />
*** 8 Rx + 8 Tx @ 250 MS/s <br />
* CG_1600 <br />
** Dual Port DPDK <br />
*** 2 Rx @ 1000 MS/s <br />
*** 2 Tx @ 1800 MS/s <br />
*** 2 Rx + 2 Tx @ 1000 MS/s <br />
<br />
All testing was done using dual 100GbE ports. For a single port, expect around half the number of channels as compared to the dual port equivalent configuration at the same streaming rate. <br />
<br />
==Guidance on SFP+ Adapters for Fiber Connectivity on NI Ettus USRP X4x0==<br />
<br />
Ettus Research currently offers direct-connect, copper cabling accessories for the NI Ettus USRP X4x0. However, it is also possible to use multi-mode fiber instead of copper connections for these devices. In this section, we will provide general guidance on the types of fiber adapters and cables that can be used with these products.<br />
<br />
The NI Ettus USRP X4x0 is compatible with most brands of SFP+ fiber adapters. In some cases, other equipment in the systems such as 1/10/100 Gigabit Ethernet switches are only compatible with specific brands of SFP+ adapters and cables. As a general rule, we recommend checking compatibility with the switches and network cards in your system before purchasing an adapter.<br />
<br />
Ettus Research does test the NI Ettus USRP X4x0 devices with the listed hardware as noted in the above section https://kb.ettus.com/X440#100_Gigabit_Ethernet<br />
<br />
==Certifications==<br />
===RoHS===<br />
As of December 1st, 2010 all Ettus Research products are RoHS compliant unless otherwise noted. More information can be found at [http://ettus.com/legal/rohs-information http://ettus.com/legal/rohs-information]<br />
<br />
===China RoHS=== <br />
'''Management Methods for Controlling Pollution Caused by Electronic Information Products Regulation'''<br />
<br />
'''Chinese Customers''' <br />
<br />
National Instruments is in compliance with the Chinese policy on the Restriction of Hazardous Substances (RoHS) used in Electronic Information Products. For more information about the National Instruments China RoHS compliance, visit [http://www.ni.com/environment/rohs_china ni.com/environment/rohs_china].<br />
<br />
==Certificate / Letter of Volatility==<br />
<br />
Certifications will soon be findable on the [https://www.ni.com/en/support/documentation/product-certifications.html NI Product Certifications lookup tool]. The Letter of Volatility we link directly for now: [[File:USRP_X440_Letter_Of_Volatility.pdf]]<br />
<br />
==Downloads==<br />
[http://files.ettus.com/manual/md_fpga.html FPGA Resources]<br />
<br />
[http://files.ettus.com/binaries/uhd_stable/ UHD Stable Binaries]<br />
<br />
[https://github.com/EttusResearch/uhd UHD Source Code on Github]<br />
<br />
<br />
[[Category:Hardware Resources]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=X440&diff=5880X4402023-09-22T20:27:57Z<p>MichaelDickens: /* Certificate / Letter of Volatility */ fix LOV to be an actual file until it is up on the NI product certification search tool</p>
<hr />
<div>== Notice ==<br />
'''When you receive a brand-new device, it is strongly recommended that you download the most recent filesystem image from the Ettus Research website and write it to the NI Ettus USRP X4x0. Instructions on downloading the latest filesystem image and writing it to the X4x0 are described in the [https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide USRP X4x0 Getting Started Guide].'''<br />
<br />
'''Note that if you are operating the device in Network Mode, then the versions of UHD running on the host computer and on the NI Ettus USRP X4x0 device must match.'''<br />
<br />
== Device Overview ==<br />
===X440===<br />
The NI Ettus USRP X440 is the widest bandwidth USRP software defined radio device. It differs architecturally from most USRPs, utilizing a direct sampling architecture that provides balun-coupled access to the ADCs and DACs on the onboard Xilinx Zynq RFSoC. This makes it well suited to use as an intermediate frequency transceiver, connecting to external front ends for applications like satellite communications (SATCOM) prototyping, SATCOM ground station deployment, and mmWave or sub-THz 6G research. USRP X440 features high channel density – 8 Tx and 8 Rx channels per device – with phase coherency across channels by sharing sample clocks. Therefore, it is ideal for applications like direction finding and radar research and prototyping. <br />
<br />
== Key Features==<br />
<br />
===X440===<br />
{|<br />
|style="vertical-align:top"|<br />
* High channel density<br />
* Reliable and fault-tolerant deployment<br />
* Stand-alone (embedded) or host-based (network streaming) operation<br />
* Fully integrated and assembled (the USRP X440 does not support swappable daughtercards)<br />
* 30 MHz to 4 GHz frequency range (tunable down to 1MHz)<br />
* Up to 1600 MHz of instantaneous bandwidth per channel<br />
* 8 RX, 8 TX in half-wide RU form factor<br />
* Xilinx Zynq-Ultrascale+ ZU28DR RFSoC<br />
* 12 bit ADC, 14 bit DAC<br />
* IQ Sample Clock rates up to 2000 MS/s<br />
* Onboard SD-FEC, DDC, DUC<br />
* Quad-core ARM Cortex-A53 up to 1.2 GHz CPU<br />
* Dual-core ARM Cortex-A5 MPCore up to 500 MHz<br />
* Two QSFP28 ports (10 Gigabit Ethernet, 100 Gigabit Ethernet)<br />
* RJ45 (1 GbE) [1]<br />
* 10 MHz Clock reference <br />
* PPS time reference<br />
* Trig In/Out Interface<br />
* Built-in GPSDO <br />
* Two FPGA Programmable GPIO Interfaces (HDMI)<br />
* 1 Type C USB host port <br />
* 1 Type C USB port (serial console, JTAG) <br />
* Watchdog timer<br />
* OpenEmbedded Linux<br />
* USRP Hardware Driver™ (UHD) open-source software API version 4.5.0 or later<br />
* RF Network on Chip (RFNoC™) FPGA development framework<br />
* Xilinx Vivado® 2021.1 Design Suite (license not included)<br />
* GNU Radio support maintained by Ettus Research™ through GR-UHD, an interface to UHD distributed by GNU Radio<br />
* [1] The RJ45 port is used for remote management of the device and does not support IQ streaming.<br />
<br />
|[[File:X440.jpg|500px|center]] <br />
<br />
<br />
|}<br />
<br />
==Hardware Specifications==<br />
===X440===<br />
* Current Hardware Revision: Module revision D and Motherboard revision F<br />
* Minimum version of UHD required: 4.5.0<br />
* USRP X440 is covered by Export Administration Regulations (EAR) NS2. Refer to [https://www.bis.doc.gov/ US Department of Commerce] for details and country chart. <br />
<br />
https://www.ni.com/docs/en-US/bundle/ettus-usrp-x440-specs/page/specs.html<br />
https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0<br />
<br />
===CAD/STP Models===<br />
<br />
If you want any CAD / STP models beyond those found here, please send an email to Ettus Support at [mailto:support@ettus.com support@ettus.com] noting your request and your use case for any such model. We will determine on a case-by-case basis whether we have any such requested model and, if so, whether to release it -- possibly requiring an NDA for any such release. Note that we do not have models on all USRPs and daughterboards, and requesting any model does not guarantee that either Ettus Research or NI will honor any such request.<br />
<br />
==FPGA==<br />
===FPGA User Modifications===<br />
The Verilog code for the FPGA in the NI Ettus USRP X4x0 is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Specifically, changing the I/O interface of the FPGA in any way, or modifying the pin and timing constraint files, could result in physical damage to other components on the motherboard, external to the FPGA, and doing this will void the warranty. Also, even if the PCIe interface is not being used, you cannot remove or reassign these pins in the constraint file. The constraint files should not be modified. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
<br />
==Interfaces and Connectivity==<br />
<br />
===Front Panel===<br />
<br />
====X440====<br />
<br />
[[File:x440_front_panel.jpg|500px|center]] <br />
<br />
===Rear Panel===<br />
<br />
[[File:x440_back_panel.jpg|500px|center]]<br />
<br />
===Ref Clock - 10 MHz===<br />
Using an external 10 MHz reference clock, a square wave will offer the best phase noise performance, but a sinusoid is acceptable.<br />
<br />
===PPS - Pulse Per Second===<br />
Using a PPS signal for timestamp synchronization requires a square wave signal (a typical PPS signal has a 20%-25% duty cycle) with a 5 Vpp amplitude. <br />
<br />
To test the PPS input, you can use the following tool from the UHD examples:<br />
<br />
* <code><args></code> are device address arguments (optional if only one USRP device is on your machine)<br />
<br />
cd <install-path>/lib/uhd/examples ./test_pps_input –args=<args><br />
<br />
===Front Panel GPIO===<br />
* https://files.ettus.com/manual/page_x400_gpio_api.html#x4x0gpio_fpanel<br />
<br />
====Power on state====<br />
<br />
* https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide#Autoboot<br />
* https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_rearpanelleds_power<br />
<br />
==Accessories==<br />
<br />
For datasheet, drawings, pricing, and purchasing please search for the Part Number listed below via https://www.ni.com/en-us/search.html<br />
<br />
* Dual 100 Gigabit Ethernet PCIe Interface Kit For Ettus USRP X4xx<br />
** Part Number: 788216-01<br />
<br />
* USRP X4xx Power Supply, 100-240VAC 50/60HZ 12VDC 24AMP<br />
** Part Number: 788204-01<br />
<br />
* QSFP28 To 4xSFP28 Breakout Cable, 1M<br />
** Part Number: 788214-01<br />
<br />
* QSFP28 Twinaxial Cable, 3M <br />
** Part Number: 788215-03<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 2 USRP X4xx Devices, Shoulder to Shoulder <br />
** Part Number: 788147-01<br />
<br />
* USRP X4xx Desktop Stack Accessory, Single USRP X4xx Device Fastened Buildup <br />
** Part Number: 788148-01<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 1 USRP X4xx Device, w/Surrogate Extension <br />
** Part Number: 788149-01<br />
<br />
* GPIO Communication Cable<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 1M <br />
*** Part Number: 152629-01<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 2M <br />
*** Part Number: 152629-02<br />
<br />
* SCB-19 Noise Rejecting, Shielded Aux I/O Connector <br />
** Part Number: 782444-01<br />
<br />
* Fan Replacement Cartridges<br />
** USRP X4xx Fan Cartridge Accessory, Exhaust<br />
*** Part Number: 788164-01<br />
** USRP X4xx Fan Cartridge Accessory, Intake<br />
*** Part Number: 788165-01<br />
<br />
==10 Gigabit Ethernet==<br />
'''Recommended 10 Gigabit Ethernet Cards'''<br />
* Dual 10 Gigabit Ethernet Interface for Ettus USRP <br />
** [https://search.ni.com/nisearch/app/main/p/bot/no/ap/global/lang/en/pg/1/q/788600-01/ ni.com part number 788600-01]<br />
* Intel X710-DA2<br />
** [http://ark.intel.com/products/83964/Intel-Ethernet-Converged-Network-Adapter-X710-DA2 Intel® Ethernet Converged Network Adapter X710-DA2 ]<br />
* Intel X710-DA4<br />
** [http://ark.intel.com/products/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4 Intel® Ethernet Converged Network Adapter X710-DA4 ]<br />
<br />
==100 Gigabit Ethernet==<br />
<br />
===X440===<br />
<br />
* Requires UHD 4.5 or later: https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0 <br />
* 100GbE Streaming only supports Linux Hosts <br />
<br />
'''Recommended 100 Gigabit Ethernet Cards'''<br />
* Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16))<br />
<br />
'''Recommended 100 Gigabit Ethernet Cables'''<br />
* Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N<br />
** Shorter length variants also recommended<br />
<br />
'''Recommended Host PC'''<br />
* At least 32 CPU Cores<br />
* At least 64 GB RAM<br />
* Ubuntu 20.04 (5.13.0-44-generic)<br />
<br />
''' Validated Hardware and Software Configuration Examples '''<br />
* Ubuntu 20.04 (5.13.0-44-generic kernel), DPDK 20.11, with AMD Ryzen Threadripper 3960X 24-Core Processor - 48 CPU - 3.6 GHz CPU freq - 64 GB RAM. Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16)). Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N cables.<br />
<br />
''' Data Throughput Rates '''<br />
<br />
Testing was completed with the following conditions<br />
* Hardware and Software Configurations listed above<br />
* CPU configured for performance mode: https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#CPU_Governor<br />
* DPDK Setup: https://files.ettus.com/manual/page_dpdk.html and https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#UHD_4.0<br />
* Enabling Tx pause Frames on X4x0 for the SFP port(s) utilized for streaming: https://files.ettus.com/manual/page_transport.html#transport_udp_linux<br />
** <code>ethtool -A sfp0 tx on</code><br />
** <code>ethtool -A sfp1 tx on</code><br />
* uhd.conf: See https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
Executing [https://github.com/EttusResearch/uhd/blob/UHD-4.5/host/examples/benchmark_rate.cpp benchmark_rate ] over multiple iterations as well as over an extended continuous time period (>12 Hours) without data loss resulted in the following maximum rates and channel counts<br />
* Usage of the <code>--priority</code> argument set to <code>high</code> in benchmark_rate which requires benchmark rate to be executed with root privileges via <code>sudo</code><br />
* Usage of single versus multiple threads in the benchmark_rate utility - controlled by using the <code>--multi_streamer</code> argument. Specifying this argument assigns one thread per channel being streamed.<br />
* Utilizing the CG_400 and CG_1600 bitfile.<br />
<br />
* CG_400 <br />
** Dual Port DPDK <br />
*** 6 Rx @ 500 MS/s <br />
*** 8 Rx @ 400 MS/s <br />
*** 6 Tx @ 500 MS/s <br />
*** 8 Tx @ 450 MS/s <br />
*** 4 Rx + 4 Tx @ 500 MS/s <br />
*** 8 Rx + 8 Tx @ 250 MS/s <br />
* CG_1600 <br />
** Dual Port DPDK <br />
*** 2 Rx @ 1000 MS/s <br />
*** 2 Tx @ 1800 MS/s <br />
*** 2 Rx + 2 Tx @ 1000 MS/s <br />
<br />
All testing was done using dual 100GbE ports. For a single port, expect around half the number of channels as compared to the dual port equivalent configuration at the same streaming rate. <br />
<br />
==Guidance on SFP+ Adapters for Fiber Connectivity on NI Ettus USRP X4x0==<br />
<br />
Ettus Research currently offers direct-connect, copper cabling accessories for the NI Ettus USRP X4x0. However, it is also possible to use multi-mode fiber instead of copper connections for these devices. In this section, we will provide general guidance on the types of fiber adapters and cables that can be used with these products.<br />
<br />
The NI Ettus USRP X4x0 is compatible with most brands of SFP+ fiber adapters. In some cases, other equipment in the systems such as 1/10/100 Gigabit Ethernet switches are only compatible with specific brands of SFP+ adapters and cables. As a general rule, we recommend checking compatibility with the switches and network cards in your system before purchasing an adapter.<br />
<br />
Ettus Research does test the NI Ettus USRP X4x0 devices with the listed hardware as noted in the above section https://kb.ettus.com/X440#100_Gigabit_Ethernet<br />
<br />
==Certifications==<br />
===RoHS===<br />
As of December 1st, 2010 all Ettus Research products are RoHS compliant unless otherwise noted. More information can be found at [http://ettus.com/legal/rohs-information http://ettus.com/legal/rohs-information]<br />
<br />
===China RoHS=== <br />
'''Management Methods for Controlling Pollution Caused by Electronic Information Products Regulation'''<br />
<br />
'''Chinese Customers''' <br />
<br />
National Instruments is in compliance with the Chinese policy on the Restriction of Hazardous Substances (RoHS) used in Electronic Information Products. For more information about the National Instruments China RoHS compliance, visit [http://www.ni.com/environment/rohs_china ni.com/environment/rohs_china].<br />
<br />
==Certificate / Letter of Volatility==<br />
<br />
Certification will soon be findable on the [https://www.ni.com/en/support/documentation/product-certifications.html NI Product Certifications lookup tool]. For now we link to it directly [[File:USRP_X440_Letter_Of_Volatility.pdf]] .<br />
<br />
==Downloads==<br />
[http://files.ettus.com/manual/md_fpga.html FPGA Resources]<br />
<br />
[http://files.ettus.com/binaries/uhd_stable/ UHD Stable Binaries]<br />
<br />
[https://github.com/EttusResearch/uhd UHD Source Code on Github]<br />
<br />
<br />
[[Category:Hardware Resources]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=File:USRP_X440_Letter_Of_Volatility.pdf&diff=5879File:USRP X440 Letter Of Volatility.pdf2023-09-22T20:27:21Z<p>MichaelDickens: </p>
<hr />
<div></div>MichaelDickenshttps://kb.ettus.com/index.php?title=UHD_Device_Eraser_and_Certificates_of_Volatility&diff=5878UHD Device Eraser and Certificates of Volatility2023-09-22T20:24:15Z<p>MichaelDickens: /* USRPs */ add X440 LOV</p>
<hr />
<div>==Application Note Number and Author==<br />
'''AN-111''' by Michael Dickens<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-04-07<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
==Abstract==<br />
This Application Note provides an overview of the UHD Device Eraser utility as well as links to the Certificates of Volatility for all Ettus products.<br />
<br />
==UHD Device Eraser Utility==<br />
<br />
The ''uhd_device_eraser'' utility overwrites all non-volatile memory from any Ettus device supported by it: most USRPs, OctoClock, and daughterboards.<br />
<br />
Note that any ''volatile'' memory on the device is cleared simply by removing all power from the device. This utility overwrites information that persist after removing all power, that stored in ''non-volatile'' memory.<br />
<br />
The utility is provided as source code, and requires UHD (headers and library) to be installed as well as the UHD source code be available on the build computer. It uses CMake to build (just like UHD does) and works with any version of UHD from 3.7.2 through 3.15.<br />
<br />
This utility will be provided per your specific request via an email to [mailto:support@ettus.com support@ettus.com] . Ettus support will first require you to note receipt of our standard warnings in a first email, then once noted will provide a tarball of the utility.<br />
<br />
NOTE: This utility, because it erases settings used by your USRP for runtime functionality, will render your Ettus device unusable ("bricked"). It is entirely possible to make your Ettus device usable again ("unbrick" it), details of which are provided by Ettus support in emails as well as the top-level README of the utility tarball.<br />
<br />
==Certificates of Volatility==<br />
<br />
===OctoClock===<br />
<br />
[https://kb.ettus.com/OctoClock_CDA-2990#Certificate_of_Volatility OctoClock]<br />
<br />
===USRPs===<br />
<br />
[https://kb.ettus.com/B200/B210/B200mini/B205mini#Certificate_of_Volatility B200 / B210]<br />
<br />
[https://kb.ettus.com/Ettus_USRP_E300_Embedded_Family_Hardware_Resources#Certificate_of_Volatility E310 / E312 / E313]<br />
<br />
[https://kb.ettus.com/E320#Certificate_of_Volatility E320]<br />
<br />
[https://kb.ettus.com/N200/N210#Certificate_of_Volatility N200 / N210]<br />
<br />
[https://kb.ettus.com/N300/N310#Certificate_of_Volatility N300 / N310]<br />
<br />
[https://kb.ettus.com/N320/N321#Certificate_of_Volatility N320 / N321]<br />
<br />
[https://kb.ettus.com/X300/X310#Certificate_of_Volatility X300 / X310]<br />
<br />
[https://kb.ettus.com/X410#Certificate_.2F_Letter_of_Volatility X410]<br />
<br />
[https://kb.ettus.com/X440#Certificate_.2F_Letter_of_Volatility X440]<br />
<br />
===Daughterboards===<br />
<br />
[https://kb.ettus.com/BasicTX/BasicRX#Certificate_of_Volatility BasicRX / BasicTX]<br />
<br />
[https://kb.ettus.com/CBX#Certificate_of_Volatility CBX]<br />
<br />
[https://kb.ettus.com/LFTX/LFRX#Certificate_of_Volatility LFRX / LFTX]<br />
<br />
[https://kb.ettus.com/SBX#Certificate_of_Volatility SBX]<br />
<br />
[https://kb.ettus.com/TwinRX#Certificate_of_Volatility TwinRX]<br />
<br />
[https://kb.ettus.com/UBX#Certificate_of_Volatility UBX (40 / 160 / 160-LP)]<br />
<br />
[https://kb.ettus.com/WBX#Certificate_of_Volatility WBX]<br />
<br />
====End-Of-Life Daughterboards====<br />
<br />
[https://www.ni.com/pdf/manuals/377300a.pdf DBSRX2]<br />
<br />
[https://www.ni.com/pdf/manuals/377300a.pdf TVRX2]<br />
<br />
[[Category:Application Notes]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=X440&diff=5877X4402023-09-22T15:53:25Z<p>MichaelDickens: /* Guidance on SFP+ Adapters for Fiber Connectivity on NI Ettus USRP X4x0 */ internal linking</p>
<hr />
<div>== Notice ==<br />
'''When you receive a brand-new device, it is strongly recommended that you download the most recent filesystem image from the Ettus Research website and write it to the NI Ettus USRP X4x0. Instructions on downloading the latest filesystem image and writing it to the X4x0 are described in the [https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide USRP X4x0 Getting Started Guide].'''<br />
<br />
'''Note that if you are operating the device in Network Mode, then the versions of UHD running on the host computer and on the NI Ettus USRP X4x0 device must match.'''<br />
<br />
== Device Overview ==<br />
===X440===<br />
The NI Ettus USRP X440 is the widest bandwidth USRP software defined radio device. It differs architecturally from most USRPs, utilizing a direct sampling architecture that provides balun-coupled access to the ADCs and DACs on the onboard Xilinx Zynq RFSoC. This makes it well suited to use as an intermediate frequency transceiver, connecting to external front ends for applications like satellite communications (SATCOM) prototyping, SATCOM ground station deployment, and mmWave or sub-THz 6G research. USRP X440 features high channel density – 8 Tx and 8 Rx channels per device – with phase coherency across channels by sharing sample clocks. Therefore, it is ideal for applications like direction finding and radar research and prototyping. <br />
<br />
== Key Features==<br />
<br />
===X440===<br />
{|<br />
|style="vertical-align:top"|<br />
* High channel density<br />
* Reliable and fault-tolerant deployment<br />
* Stand-alone (embedded) or host-based (network streaming) operation<br />
* Fully integrated and assembled (the USRP X440 does not support swappable daughtercards)<br />
* 30 MHz to 4 GHz frequency range (tunable down to 1MHz)<br />
* Up to 1600 MHz of instantaneous bandwidth per channel<br />
* 8 RX, 8 TX in half-wide RU form factor<br />
* Xilinx Zynq-Ultrascale+ ZU28DR RFSoC<br />
* 12 bit ADC, 14 bit DAC<br />
* IQ Sample Clock rates up to 2000 MS/s<br />
* Onboard SD-FEC, DDC, DUC<br />
* Quad-core ARM Cortex-A53 up to 1.2 GHz CPU<br />
* Dual-core ARM Cortex-A5 MPCore up to 500 MHz<br />
* Two QSFP28 ports (10 Gigabit Ethernet, 100 Gigabit Ethernet)<br />
* RJ45 (1 GbE) [1]<br />
* 10 MHz Clock reference <br />
* PPS time reference<br />
* Trig In/Out Interface<br />
* Built-in GPSDO <br />
* Two FPGA Programmable GPIO Interfaces (HDMI)<br />
* 1 Type C USB host port <br />
* 1 Type C USB port (serial console, JTAG) <br />
* Watchdog timer<br />
* OpenEmbedded Linux<br />
* USRP Hardware Driver™ (UHD) open-source software API version 4.5.0 or later<br />
* RF Network on Chip (RFNoC™) FPGA development framework<br />
* Xilinx Vivado® 2021.1 Design Suite (license not included)<br />
* GNU Radio support maintained by Ettus Research™ through GR-UHD, an interface to UHD distributed by GNU Radio<br />
* [1] The RJ45 port is used for remote management of the device and does not support IQ streaming.<br />
<br />
|[[File:X440.jpg|500px|center]] <br />
<br />
<br />
|}<br />
<br />
==Hardware Specifications==<br />
===X440===<br />
* Current Hardware Revision: Module revision D and Motherboard revision F<br />
* Minimum version of UHD required: 4.5.0<br />
* USRP X440 is covered by Export Administration Regulations (EAR) NS2. Refer to [https://www.bis.doc.gov/ US Department of Commerce] for details and country chart. <br />
<br />
https://www.ni.com/docs/en-US/bundle/ettus-usrp-x440-specs/page/specs.html<br />
https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0<br />
<br />
===CAD/STP Models===<br />
<br />
If you want any CAD / STP models beyond those found here, please send an email to Ettus Support at [mailto:support@ettus.com support@ettus.com] noting your request and your use case for any such model. We will determine on a case-by-case basis whether we have any such requested model and, if so, whether to release it -- possibly requiring an NDA for any such release. Note that we do not have models on all USRPs and daughterboards, and requesting any model does not guarantee that either Ettus Research or NI will honor any such request.<br />
<br />
==FPGA==<br />
===FPGA User Modifications===<br />
The Verilog code for the FPGA in the NI Ettus USRP X4x0 is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Specifically, changing the I/O interface of the FPGA in any way, or modifying the pin and timing constraint files, could result in physical damage to other components on the motherboard, external to the FPGA, and doing this will void the warranty. Also, even if the PCIe interface is not being used, you cannot remove or reassign these pins in the constraint file. The constraint files should not be modified. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
<br />
==Interfaces and Connectivity==<br />
<br />
===Front Panel===<br />
<br />
====X440====<br />
<br />
[[File:x440_front_panel.jpg|500px|center]] <br />
<br />
===Rear Panel===<br />
<br />
[[File:x440_back_panel.jpg|500px|center]]<br />
<br />
===Ref Clock - 10 MHz===<br />
Using an external 10 MHz reference clock, a square wave will offer the best phase noise performance, but a sinusoid is acceptable.<br />
<br />
===PPS - Pulse Per Second===<br />
Using a PPS signal for timestamp synchronization requires a square wave signal (a typical PPS signal has a 20%-25% duty cycle) with a 5 Vpp amplitude. <br />
<br />
To test the PPS input, you can use the following tool from the UHD examples:<br />
<br />
* <code><args></code> are device address arguments (optional if only one USRP device is on your machine)<br />
<br />
cd <install-path>/lib/uhd/examples ./test_pps_input –args=<args><br />
<br />
===Front Panel GPIO===<br />
* https://files.ettus.com/manual/page_x400_gpio_api.html#x4x0gpio_fpanel<br />
<br />
====Power on state====<br />
<br />
* https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide#Autoboot<br />
* https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_rearpanelleds_power<br />
<br />
==Accessories==<br />
<br />
For datasheet, drawings, pricing, and purchasing please search for the Part Number listed below via https://www.ni.com/en-us/search.html<br />
<br />
* Dual 100 Gigabit Ethernet PCIe Interface Kit For Ettus USRP X4xx<br />
** Part Number: 788216-01<br />
<br />
* USRP X4xx Power Supply, 100-240VAC 50/60HZ 12VDC 24AMP<br />
** Part Number: 788204-01<br />
<br />
* QSFP28 To 4xSFP28 Breakout Cable, 1M<br />
** Part Number: 788214-01<br />
<br />
* QSFP28 Twinaxial Cable, 3M <br />
** Part Number: 788215-03<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 2 USRP X4xx Devices, Shoulder to Shoulder <br />
** Part Number: 788147-01<br />
<br />
* USRP X4xx Desktop Stack Accessory, Single USRP X4xx Device Fastened Buildup <br />
** Part Number: 788148-01<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 1 USRP X4xx Device, w/Surrogate Extension <br />
** Part Number: 788149-01<br />
<br />
* GPIO Communication Cable<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 1M <br />
*** Part Number: 152629-01<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 2M <br />
*** Part Number: 152629-02<br />
<br />
* SCB-19 Noise Rejecting, Shielded Aux I/O Connector <br />
** Part Number: 782444-01<br />
<br />
* Fan Replacement Cartridges<br />
** USRP X4xx Fan Cartridge Accessory, Exhaust<br />
*** Part Number: 788164-01<br />
** USRP X4xx Fan Cartridge Accessory, Intake<br />
*** Part Number: 788165-01<br />
<br />
==10 Gigabit Ethernet==<br />
'''Recommended 10 Gigabit Ethernet Cards'''<br />
* Dual 10 Gigabit Ethernet Interface for Ettus USRP <br />
** [https://search.ni.com/nisearch/app/main/p/bot/no/ap/global/lang/en/pg/1/q/788600-01/ ni.com part number 788600-01]<br />
* Intel X710-DA2<br />
** [http://ark.intel.com/products/83964/Intel-Ethernet-Converged-Network-Adapter-X710-DA2 Intel® Ethernet Converged Network Adapter X710-DA2 ]<br />
* Intel X710-DA4<br />
** [http://ark.intel.com/products/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4 Intel® Ethernet Converged Network Adapter X710-DA4 ]<br />
<br />
==100 Gigabit Ethernet==<br />
<br />
===X440===<br />
<br />
* Requires UHD 4.5 or later: https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0 <br />
* 100GbE Streaming only supports Linux Hosts <br />
<br />
'''Recommended 100 Gigabit Ethernet Cards'''<br />
* Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16))<br />
<br />
'''Recommended 100 Gigabit Ethernet Cables'''<br />
* Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N<br />
** Shorter length variants also recommended<br />
<br />
'''Recommended Host PC'''<br />
* At least 32 CPU Cores<br />
* At least 64 GB RAM<br />
* Ubuntu 20.04 (5.13.0-44-generic)<br />
<br />
''' Validated Hardware and Software Configuration Examples '''<br />
* Ubuntu 20.04 (5.13.0-44-generic kernel), DPDK 20.11, with AMD Ryzen Threadripper 3960X 24-Core Processor - 48 CPU - 3.6 GHz CPU freq - 64 GB RAM. Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16)). Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N cables.<br />
<br />
''' Data Throughput Rates '''<br />
<br />
Testing was completed with the following conditions<br />
* Hardware and Software Configurations listed above<br />
* CPU configured for performance mode: https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#CPU_Governor<br />
* DPDK Setup: https://files.ettus.com/manual/page_dpdk.html and https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#UHD_4.0<br />
* Enabling Tx pause Frames on X4x0 for the SFP port(s) utilized for streaming: https://files.ettus.com/manual/page_transport.html#transport_udp_linux<br />
** <code>ethtool -A sfp0 tx on</code><br />
** <code>ethtool -A sfp1 tx on</code><br />
* uhd.conf: See https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
Executing [https://github.com/EttusResearch/uhd/blob/UHD-4.5/host/examples/benchmark_rate.cpp benchmark_rate ] over multiple iterations as well as over an extended continuous time period (>12 Hours) without data loss resulted in the following maximum rates and channel counts<br />
* Usage of the <code>--priority</code> argument set to <code>high</code> in benchmark_rate which requires benchmark rate to be executed with root privileges via <code>sudo</code><br />
* Usage of single versus multiple threads in the benchmark_rate utility - controlled by using the <code>--multi_streamer</code> argument. Specifying this argument assigns one thread per channel being streamed.<br />
* Utilizing the CG_400 and CG_1600 bitfile.<br />
<br />
* CG_400 <br />
** Dual Port DPDK <br />
*** 6 Rx @ 500 MS/s <br />
*** 8 Rx @ 400 MS/s <br />
*** 6 Tx @ 500 MS/s <br />
*** 8 Tx @ 450 MS/s <br />
*** 4 Rx + 4 Tx @ 500 MS/s <br />
*** 8 Rx + 8 Tx @ 250 MS/s <br />
* CG_1600 <br />
** Dual Port DPDK <br />
*** 2 Rx @ 1000 MS/s <br />
*** 2 Tx @ 1800 MS/s <br />
*** 2 Rx + 2 Tx @ 1000 MS/s <br />
<br />
All testing was done using dual 100GbE ports. For a single port, expect around half the number of channels as compared to the dual port equivalent configuration at the same streaming rate. <br />
<br />
==Guidance on SFP+ Adapters for Fiber Connectivity on NI Ettus USRP X4x0==<br />
<br />
Ettus Research currently offers direct-connect, copper cabling accessories for the NI Ettus USRP X4x0. However, it is also possible to use multi-mode fiber instead of copper connections for these devices. In this section, we will provide general guidance on the types of fiber adapters and cables that can be used with these products.<br />
<br />
The NI Ettus USRP X4x0 is compatible with most brands of SFP+ fiber adapters. In some cases, other equipment in the systems such as 1/10/100 Gigabit Ethernet switches are only compatible with specific brands of SFP+ adapters and cables. As a general rule, we recommend checking compatibility with the switches and network cards in your system before purchasing an adapter.<br />
<br />
Ettus Research does test the NI Ettus USRP X4x0 devices with the listed hardware as noted in the above section https://kb.ettus.com/X440#100_Gigabit_Ethernet<br />
<br />
==Certifications==<br />
===RoHS===<br />
As of December 1st, 2010 all Ettus Research products are RoHS compliant unless otherwise noted. More information can be found at [http://ettus.com/legal/rohs-information http://ettus.com/legal/rohs-information]<br />
<br />
===China RoHS=== <br />
'''Management Methods for Controlling Pollution Caused by Electronic Information Products Regulation'''<br />
<br />
'''Chinese Customers''' <br />
<br />
National Instruments is in compliance with the Chinese policy on the Restriction of Hazardous Substances (RoHS) used in Electronic Information Products. For more information about the National Instruments China RoHS compliance, visit [http://www.ni.com/environment/rohs_china ni.com/environment/rohs_china].<br />
<br />
==Certificate / Letter of Volatility==<br />
<br />
Certification can be found on the [https://www.ni.com/en/support/documentation/product-certifications.html NI Product Certifications lookup tool]<br />
[https://www.ni.com/docs/en-US/bundle/ni-ettus-usrp-x410-lov/resource/378157a.pdf X410 LOV]<br />
<br />
==Downloads==<br />
[http://files.ettus.com/manual/md_fpga.html FPGA Resources]<br />
<br />
[http://files.ettus.com/binaries/uhd_stable/ UHD Stable Binaries]<br />
<br />
[https://github.com/EttusResearch/uhd UHD Source Code on Github]<br />
<br />
<br />
[[Category:Hardware Resources]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=X440&diff=5876X4402023-09-22T15:50:06Z<p>MichaelDickens: /* Power on state */ X410/X440</p>
<hr />
<div>== Notice ==<br />
'''When you receive a brand-new device, it is strongly recommended that you download the most recent filesystem image from the Ettus Research website and write it to the NI Ettus USRP X4x0. Instructions on downloading the latest filesystem image and writing it to the X4x0 are described in the [https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide USRP X4x0 Getting Started Guide].'''<br />
<br />
'''Note that if you are operating the device in Network Mode, then the versions of UHD running on the host computer and on the NI Ettus USRP X4x0 device must match.'''<br />
<br />
== Device Overview ==<br />
===X440===<br />
The NI Ettus USRP X440 is the widest bandwidth USRP software defined radio device. It differs architecturally from most USRPs, utilizing a direct sampling architecture that provides balun-coupled access to the ADCs and DACs on the onboard Xilinx Zynq RFSoC. This makes it well suited to use as an intermediate frequency transceiver, connecting to external front ends for applications like satellite communications (SATCOM) prototyping, SATCOM ground station deployment, and mmWave or sub-THz 6G research. USRP X440 features high channel density – 8 Tx and 8 Rx channels per device – with phase coherency across channels by sharing sample clocks. Therefore, it is ideal for applications like direction finding and radar research and prototyping. <br />
<br />
== Key Features==<br />
<br />
===X440===<br />
{|<br />
|style="vertical-align:top"|<br />
* High channel density<br />
* Reliable and fault-tolerant deployment<br />
* Stand-alone (embedded) or host-based (network streaming) operation<br />
* Fully integrated and assembled (the USRP X440 does not support swappable daughtercards)<br />
* 30 MHz to 4 GHz frequency range (tunable down to 1MHz)<br />
* Up to 1600 MHz of instantaneous bandwidth per channel<br />
* 8 RX, 8 TX in half-wide RU form factor<br />
* Xilinx Zynq-Ultrascale+ ZU28DR RFSoC<br />
* 12 bit ADC, 14 bit DAC<br />
* IQ Sample Clock rates up to 2000 MS/s<br />
* Onboard SD-FEC, DDC, DUC<br />
* Quad-core ARM Cortex-A53 up to 1.2 GHz CPU<br />
* Dual-core ARM Cortex-A5 MPCore up to 500 MHz<br />
* Two QSFP28 ports (10 Gigabit Ethernet, 100 Gigabit Ethernet)<br />
* RJ45 (1 GbE) [1]<br />
* 10 MHz Clock reference <br />
* PPS time reference<br />
* Trig In/Out Interface<br />
* Built-in GPSDO <br />
* Two FPGA Programmable GPIO Interfaces (HDMI)<br />
* 1 Type C USB host port <br />
* 1 Type C USB port (serial console, JTAG) <br />
* Watchdog timer<br />
* OpenEmbedded Linux<br />
* USRP Hardware Driver™ (UHD) open-source software API version 4.5.0 or later<br />
* RF Network on Chip (RFNoC™) FPGA development framework<br />
* Xilinx Vivado® 2021.1 Design Suite (license not included)<br />
* GNU Radio support maintained by Ettus Research™ through GR-UHD, an interface to UHD distributed by GNU Radio<br />
* [1] The RJ45 port is used for remote management of the device and does not support IQ streaming.<br />
<br />
|[[File:X440.jpg|500px|center]] <br />
<br />
<br />
|}<br />
<br />
==Hardware Specifications==<br />
===X440===<br />
* Current Hardware Revision: Module revision D and Motherboard revision F<br />
* Minimum version of UHD required: 4.5.0<br />
* USRP X440 is covered by Export Administration Regulations (EAR) NS2. Refer to [https://www.bis.doc.gov/ US Department of Commerce] for details and country chart. <br />
<br />
https://www.ni.com/docs/en-US/bundle/ettus-usrp-x440-specs/page/specs.html<br />
https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0<br />
<br />
===CAD/STP Models===<br />
<br />
If you want any CAD / STP models beyond those found here, please send an email to Ettus Support at [mailto:support@ettus.com support@ettus.com] noting your request and your use case for any such model. We will determine on a case-by-case basis whether we have any such requested model and, if so, whether to release it -- possibly requiring an NDA for any such release. Note that we do not have models on all USRPs and daughterboards, and requesting any model does not guarantee that either Ettus Research or NI will honor any such request.<br />
<br />
==FPGA==<br />
===FPGA User Modifications===<br />
The Verilog code for the FPGA in the NI Ettus USRP X4x0 is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Specifically, changing the I/O interface of the FPGA in any way, or modifying the pin and timing constraint files, could result in physical damage to other components on the motherboard, external to the FPGA, and doing this will void the warranty. Also, even if the PCIe interface is not being used, you cannot remove or reassign these pins in the constraint file. The constraint files should not be modified. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
<br />
==Interfaces and Connectivity==<br />
<br />
===Front Panel===<br />
<br />
====X440====<br />
<br />
[[File:x440_front_panel.jpg|500px|center]] <br />
<br />
===Rear Panel===<br />
<br />
[[File:x440_back_panel.jpg|500px|center]]<br />
<br />
===Ref Clock - 10 MHz===<br />
Using an external 10 MHz reference clock, a square wave will offer the best phase noise performance, but a sinusoid is acceptable.<br />
<br />
===PPS - Pulse Per Second===<br />
Using a PPS signal for timestamp synchronization requires a square wave signal (a typical PPS signal has a 20%-25% duty cycle) with a 5 Vpp amplitude. <br />
<br />
To test the PPS input, you can use the following tool from the UHD examples:<br />
<br />
* <code><args></code> are device address arguments (optional if only one USRP device is on your machine)<br />
<br />
cd <install-path>/lib/uhd/examples ./test_pps_input –args=<args><br />
<br />
===Front Panel GPIO===<br />
* https://files.ettus.com/manual/page_x400_gpio_api.html#x4x0gpio_fpanel<br />
<br />
====Power on state====<br />
<br />
* https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide#Autoboot<br />
* https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_rearpanelleds_power<br />
<br />
==Accessories==<br />
<br />
For datasheet, drawings, pricing, and purchasing please search for the Part Number listed below via https://www.ni.com/en-us/search.html<br />
<br />
* Dual 100 Gigabit Ethernet PCIe Interface Kit For Ettus USRP X4xx<br />
** Part Number: 788216-01<br />
<br />
* USRP X4xx Power Supply, 100-240VAC 50/60HZ 12VDC 24AMP<br />
** Part Number: 788204-01<br />
<br />
* QSFP28 To 4xSFP28 Breakout Cable, 1M<br />
** Part Number: 788214-01<br />
<br />
* QSFP28 Twinaxial Cable, 3M <br />
** Part Number: 788215-03<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 2 USRP X4xx Devices, Shoulder to Shoulder <br />
** Part Number: 788147-01<br />
<br />
* USRP X4xx Desktop Stack Accessory, Single USRP X4xx Device Fastened Buildup <br />
** Part Number: 788148-01<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 1 USRP X4xx Device, w/Surrogate Extension <br />
** Part Number: 788149-01<br />
<br />
* GPIO Communication Cable<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 1M <br />
*** Part Number: 152629-01<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 2M <br />
*** Part Number: 152629-02<br />
<br />
* SCB-19 Noise Rejecting, Shielded Aux I/O Connector <br />
** Part Number: 782444-01<br />
<br />
* Fan Replacement Cartridges<br />
** USRP X4xx Fan Cartridge Accessory, Exhaust<br />
*** Part Number: 788164-01<br />
** USRP X4xx Fan Cartridge Accessory, Intake<br />
*** Part Number: 788165-01<br />
<br />
==10 Gigabit Ethernet==<br />
'''Recommended 10 Gigabit Ethernet Cards'''<br />
* Dual 10 Gigabit Ethernet Interface for Ettus USRP <br />
** [https://search.ni.com/nisearch/app/main/p/bot/no/ap/global/lang/en/pg/1/q/788600-01/ ni.com part number 788600-01]<br />
* Intel X710-DA2<br />
** [http://ark.intel.com/products/83964/Intel-Ethernet-Converged-Network-Adapter-X710-DA2 Intel® Ethernet Converged Network Adapter X710-DA2 ]<br />
* Intel X710-DA4<br />
** [http://ark.intel.com/products/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4 Intel® Ethernet Converged Network Adapter X710-DA4 ]<br />
<br />
==100 Gigabit Ethernet==<br />
<br />
===X440===<br />
<br />
* Requires UHD 4.5 or later: https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0 <br />
* 100GbE Streaming only supports Linux Hosts <br />
<br />
'''Recommended 100 Gigabit Ethernet Cards'''<br />
* Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16))<br />
<br />
'''Recommended 100 Gigabit Ethernet Cables'''<br />
* Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N<br />
** Shorter length variants also recommended<br />
<br />
'''Recommended Host PC'''<br />
* At least 32 CPU Cores<br />
* At least 64 GB RAM<br />
* Ubuntu 20.04 (5.13.0-44-generic)<br />
<br />
''' Validated Hardware and Software Configuration Examples '''<br />
* Ubuntu 20.04 (5.13.0-44-generic kernel), DPDK 20.11, with AMD Ryzen Threadripper 3960X 24-Core Processor - 48 CPU - 3.6 GHz CPU freq - 64 GB RAM. Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16)). Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N cables.<br />
<br />
''' Data Throughput Rates '''<br />
<br />
Testing was completed with the following conditions<br />
* Hardware and Software Configurations listed above<br />
* CPU configured for performance mode: https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#CPU_Governor<br />
* DPDK Setup: https://files.ettus.com/manual/page_dpdk.html and https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#UHD_4.0<br />
* Enabling Tx pause Frames on X4x0 for the SFP port(s) utilized for streaming: https://files.ettus.com/manual/page_transport.html#transport_udp_linux<br />
** <code>ethtool -A sfp0 tx on</code><br />
** <code>ethtool -A sfp1 tx on</code><br />
* uhd.conf: See https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
Executing [https://github.com/EttusResearch/uhd/blob/UHD-4.5/host/examples/benchmark_rate.cpp benchmark_rate ] over multiple iterations as well as over an extended continuous time period (>12 Hours) without data loss resulted in the following maximum rates and channel counts<br />
* Usage of the <code>--priority</code> argument set to <code>high</code> in benchmark_rate which requires benchmark rate to be executed with root privileges via <code>sudo</code><br />
* Usage of single versus multiple threads in the benchmark_rate utility - controlled by using the <code>--multi_streamer</code> argument. Specifying this argument assigns one thread per channel being streamed.<br />
* Utilizing the CG_400 and CG_1600 bitfile.<br />
<br />
* CG_400 <br />
** Dual Port DPDK <br />
*** 6 Rx @ 500 MS/s <br />
*** 8 Rx @ 400 MS/s <br />
*** 6 Tx @ 500 MS/s <br />
*** 8 Tx @ 450 MS/s <br />
*** 4 Rx + 4 Tx @ 500 MS/s <br />
*** 8 Rx + 8 Tx @ 250 MS/s <br />
* CG_1600 <br />
** Dual Port DPDK <br />
*** 2 Rx @ 1000 MS/s <br />
*** 2 Tx @ 1800 MS/s <br />
*** 2 Rx + 2 Tx @ 1000 MS/s <br />
<br />
All testing was done using dual 100GbE ports. For a single port, expect around half the number of channels as compared to the dual port equivalent configuration at the same streaming rate. <br />
<br />
==Guidance on SFP+ Adapters for Fiber Connectivity on NI Ettus USRP X4x0==<br />
<br />
Ettus Research currently offers direct-connect, copper cabling accessories for the NI Ettus USRP X4x0. However, it is also possible to use multi-mode fiber instead of copper connections for these devices. In this section, we will provide general guidance on the types of fiber adapters and cables that can be used with these products.<br />
<br />
The NI Ettus USRP X4x0 is compatible with most brands of SFP+ fiber adapters. In some cases, other equipment in the systems such as 1/10/100 Gigabit Ethernet switches are only compatible with specific brands of SFP+ adapters and cables. As a general rule, we recommend checking compatibility with the switches and network cards in your system before purchasing an adapter.<br />
<br />
Ettus Research does test the NI Ettus USRP X4x0 devices with the listed hardware as noted in the above section https://kb.ettus.com/X410#100_Gigabit_Ethernet<br />
<br />
==Certifications==<br />
===RoHS===<br />
As of December 1st, 2010 all Ettus Research products are RoHS compliant unless otherwise noted. More information can be found at [http://ettus.com/legal/rohs-information http://ettus.com/legal/rohs-information]<br />
<br />
===China RoHS=== <br />
'''Management Methods for Controlling Pollution Caused by Electronic Information Products Regulation'''<br />
<br />
'''Chinese Customers''' <br />
<br />
National Instruments is in compliance with the Chinese policy on the Restriction of Hazardous Substances (RoHS) used in Electronic Information Products. For more information about the National Instruments China RoHS compliance, visit [http://www.ni.com/environment/rohs_china ni.com/environment/rohs_china].<br />
<br />
==Certificate / Letter of Volatility==<br />
<br />
Certification can be found on the [https://www.ni.com/en/support/documentation/product-certifications.html NI Product Certifications lookup tool]<br />
[https://www.ni.com/docs/en-US/bundle/ni-ettus-usrp-x410-lov/resource/378157a.pdf X410 LOV]<br />
<br />
==Downloads==<br />
[http://files.ettus.com/manual/md_fpga.html FPGA Resources]<br />
<br />
[http://files.ettus.com/binaries/uhd_stable/ UHD Stable Binaries]<br />
<br />
[https://github.com/EttusResearch/uhd UHD Source Code on Github]<br />
<br />
<br />
[[Category:Hardware Resources]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=X440&diff=5875X4402023-09-22T15:48:59Z<p>MichaelDickens: /* Notice */ X410/X440</p>
<hr />
<div>== Notice ==<br />
'''When you receive a brand-new device, it is strongly recommended that you download the most recent filesystem image from the Ettus Research website and write it to the NI Ettus USRP X4x0. Instructions on downloading the latest filesystem image and writing it to the X4x0 are described in the [https://kb.ettus.com/USRP_X410/X440_Getting_Started_Guide USRP X4x0 Getting Started Guide].'''<br />
<br />
'''Note that if you are operating the device in Network Mode, then the versions of UHD running on the host computer and on the NI Ettus USRP X4x0 device must match.'''<br />
<br />
== Device Overview ==<br />
===X440===<br />
The NI Ettus USRP X440 is the widest bandwidth USRP software defined radio device. It differs architecturally from most USRPs, utilizing a direct sampling architecture that provides balun-coupled access to the ADCs and DACs on the onboard Xilinx Zynq RFSoC. This makes it well suited to use as an intermediate frequency transceiver, connecting to external front ends for applications like satellite communications (SATCOM) prototyping, SATCOM ground station deployment, and mmWave or sub-THz 6G research. USRP X440 features high channel density – 8 Tx and 8 Rx channels per device – with phase coherency across channels by sharing sample clocks. Therefore, it is ideal for applications like direction finding and radar research and prototyping. <br />
<br />
== Key Features==<br />
<br />
===X440===<br />
{|<br />
|style="vertical-align:top"|<br />
* High channel density<br />
* Reliable and fault-tolerant deployment<br />
* Stand-alone (embedded) or host-based (network streaming) operation<br />
* Fully integrated and assembled (the USRP X440 does not support swappable daughtercards)<br />
* 30 MHz to 4 GHz frequency range (tunable down to 1MHz)<br />
* Up to 1600 MHz of instantaneous bandwidth per channel<br />
* 8 RX, 8 TX in half-wide RU form factor<br />
* Xilinx Zynq-Ultrascale+ ZU28DR RFSoC<br />
* 12 bit ADC, 14 bit DAC<br />
* IQ Sample Clock rates up to 2000 MS/s<br />
* Onboard SD-FEC, DDC, DUC<br />
* Quad-core ARM Cortex-A53 up to 1.2 GHz CPU<br />
* Dual-core ARM Cortex-A5 MPCore up to 500 MHz<br />
* Two QSFP28 ports (10 Gigabit Ethernet, 100 Gigabit Ethernet)<br />
* RJ45 (1 GbE) [1]<br />
* 10 MHz Clock reference <br />
* PPS time reference<br />
* Trig In/Out Interface<br />
* Built-in GPSDO <br />
* Two FPGA Programmable GPIO Interfaces (HDMI)<br />
* 1 Type C USB host port <br />
* 1 Type C USB port (serial console, JTAG) <br />
* Watchdog timer<br />
* OpenEmbedded Linux<br />
* USRP Hardware Driver™ (UHD) open-source software API version 4.5.0 or later<br />
* RF Network on Chip (RFNoC™) FPGA development framework<br />
* Xilinx Vivado® 2021.1 Design Suite (license not included)<br />
* GNU Radio support maintained by Ettus Research™ through GR-UHD, an interface to UHD distributed by GNU Radio<br />
* [1] The RJ45 port is used for remote management of the device and does not support IQ streaming.<br />
<br />
|[[File:X440.jpg|500px|center]] <br />
<br />
<br />
|}<br />
<br />
==Hardware Specifications==<br />
===X440===<br />
* Current Hardware Revision: Module revision D and Motherboard revision F<br />
* Minimum version of UHD required: 4.5.0<br />
* USRP X440 is covered by Export Administration Regulations (EAR) NS2. Refer to [https://www.bis.doc.gov/ US Department of Commerce] for details and country chart. <br />
<br />
https://www.ni.com/docs/en-US/bundle/ettus-usrp-x440-specs/page/specs.html<br />
https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0<br />
<br />
===CAD/STP Models===<br />
<br />
If you want any CAD / STP models beyond those found here, please send an email to Ettus Support at [mailto:support@ettus.com support@ettus.com] noting your request and your use case for any such model. We will determine on a case-by-case basis whether we have any such requested model and, if so, whether to release it -- possibly requiring an NDA for any such release. Note that we do not have models on all USRPs and daughterboards, and requesting any model does not guarantee that either Ettus Research or NI will honor any such request.<br />
<br />
==FPGA==<br />
===FPGA User Modifications===<br />
The Verilog code for the FPGA in the NI Ettus USRP X4x0 is open-source, and users are free to modify and customize it for their needs. However, certain modifications may result in either bricking the device, or even in physical damage to the unit. Specifically, changing the I/O interface of the FPGA in any way, or modifying the pin and timing constraint files, could result in physical damage to other components on the motherboard, external to the FPGA, and doing this will void the warranty. Also, even if the PCIe interface is not being used, you cannot remove or reassign these pins in the constraint file. The constraint files should not be modified. Please note that modifications to the FPGA are made at the risk of the user, and may not be covered by the warranty of the device.<br />
<br />
==Interfaces and Connectivity==<br />
<br />
===Front Panel===<br />
<br />
====X440====<br />
<br />
[[File:x440_front_panel.jpg|500px|center]] <br />
<br />
===Rear Panel===<br />
<br />
[[File:x440_back_panel.jpg|500px|center]]<br />
<br />
===Ref Clock - 10 MHz===<br />
Using an external 10 MHz reference clock, a square wave will offer the best phase noise performance, but a sinusoid is acceptable.<br />
<br />
===PPS - Pulse Per Second===<br />
Using a PPS signal for timestamp synchronization requires a square wave signal (a typical PPS signal has a 20%-25% duty cycle) with a 5 Vpp amplitude. <br />
<br />
To test the PPS input, you can use the following tool from the UHD examples:<br />
<br />
* <code><args></code> are device address arguments (optional if only one USRP device is on your machine)<br />
<br />
cd <install-path>/lib/uhd/examples ./test_pps_input –args=<args><br />
<br />
===Front Panel GPIO===<br />
* https://files.ettus.com/manual/page_x400_gpio_api.html#x4x0gpio_fpanel<br />
<br />
====Power on state====<br />
<br />
* https://kb.ettus.com/USRP_X410_Getting_Started_Guide#Autoboot<br />
* https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_rearpanelleds_power<br />
<br />
==Accessories==<br />
<br />
For datasheet, drawings, pricing, and purchasing please search for the Part Number listed below via https://www.ni.com/en-us/search.html<br />
<br />
* Dual 100 Gigabit Ethernet PCIe Interface Kit For Ettus USRP X4xx<br />
** Part Number: 788216-01<br />
<br />
* USRP X4xx Power Supply, 100-240VAC 50/60HZ 12VDC 24AMP<br />
** Part Number: 788204-01<br />
<br />
* QSFP28 To 4xSFP28 Breakout Cable, 1M<br />
** Part Number: 788214-01<br />
<br />
* QSFP28 Twinaxial Cable, 3M <br />
** Part Number: 788215-03<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 2 USRP X4xx Devices, Shoulder to Shoulder <br />
** Part Number: 788147-01<br />
<br />
* USRP X4xx Desktop Stack Accessory, Single USRP X4xx Device Fastened Buildup <br />
** Part Number: 788148-01<br />
<br />
* USRP X4xx 19" Rack Mount Accessory, 1U, 1 USRP X4xx Device, w/Surrogate Extension <br />
** Part Number: 788149-01<br />
<br />
* GPIO Communication Cable<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 1M <br />
*** Part Number: 152629-01<br />
** SHH19-H19-AUX Shielded Single-Ended Cable, 2M <br />
*** Part Number: 152629-02<br />
<br />
* SCB-19 Noise Rejecting, Shielded Aux I/O Connector <br />
** Part Number: 782444-01<br />
<br />
* Fan Replacement Cartridges<br />
** USRP X4xx Fan Cartridge Accessory, Exhaust<br />
*** Part Number: 788164-01<br />
** USRP X4xx Fan Cartridge Accessory, Intake<br />
*** Part Number: 788165-01<br />
<br />
==10 Gigabit Ethernet==<br />
'''Recommended 10 Gigabit Ethernet Cards'''<br />
* Dual 10 Gigabit Ethernet Interface for Ettus USRP <br />
** [https://search.ni.com/nisearch/app/main/p/bot/no/ap/global/lang/en/pg/1/q/788600-01/ ni.com part number 788600-01]<br />
* Intel X710-DA2<br />
** [http://ark.intel.com/products/83964/Intel-Ethernet-Converged-Network-Adapter-X710-DA2 Intel® Ethernet Converged Network Adapter X710-DA2 ]<br />
* Intel X710-DA4<br />
** [http://ark.intel.com/products/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4 Intel® Ethernet Converged Network Adapter X710-DA4 ]<br />
<br />
==100 Gigabit Ethernet==<br />
<br />
===X440===<br />
<br />
* Requires UHD 4.5 or later: https://github.com/EttusResearch/uhd/releases/tag/v4.5.0.0 <br />
* 100GbE Streaming only supports Linux Hosts <br />
<br />
'''Recommended 100 Gigabit Ethernet Cards'''<br />
* Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16))<br />
<br />
'''Recommended 100 Gigabit Ethernet Cables'''<br />
* Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N<br />
** Shorter length variants also recommended<br />
<br />
'''Recommended Host PC'''<br />
* At least 32 CPU Cores<br />
* At least 64 GB RAM<br />
* Ubuntu 20.04 (5.13.0-44-generic)<br />
<br />
''' Validated Hardware and Software Configuration Examples '''<br />
* Ubuntu 20.04 (5.13.0-44-generic kernel), DPDK 20.11, with AMD Ryzen Threadripper 3960X 24-Core Processor - 48 CPU - 3.6 GHz CPU freq - 64 GB RAM. Mellanox/NVIDIA ConnectX-5 EX 100 GbE NIC (MCX516A-CDAT (PCIe Gen4 x16)). Mellanox/NVIDIA 3m QSFP28 MCP1600-C003E26N cables.<br />
<br />
''' Data Throughput Rates '''<br />
<br />
Testing was completed with the following conditions<br />
* Hardware and Software Configurations listed above<br />
* CPU configured for performance mode: https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#CPU_Governor<br />
* DPDK Setup: https://files.ettus.com/manual/page_dpdk.html and https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#UHD_4.0<br />
* Enabling Tx pause Frames on X4x0 for the SFP port(s) utilized for streaming: https://files.ettus.com/manual/page_transport.html#transport_udp_linux<br />
** <code>ethtool -A sfp0 tx on</code><br />
** <code>ethtool -A sfp1 tx on</code><br />
* uhd.conf: See https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config<br />
<br />
Executing [https://github.com/EttusResearch/uhd/blob/UHD-4.5/host/examples/benchmark_rate.cpp benchmark_rate ] over multiple iterations as well as over an extended continuous time period (>12 Hours) without data loss resulted in the following maximum rates and channel counts<br />
* Usage of the <code>--priority</code> argument set to <code>high</code> in benchmark_rate which requires benchmark rate to be executed with root privileges via <code>sudo</code><br />
* Usage of single versus multiple threads in the benchmark_rate utility - controlled by using the <code>--multi_streamer</code> argument. Specifying this argument assigns one thread per channel being streamed.<br />
* Utilizing the CG_400 and CG_1600 bitfile.<br />
<br />
* CG_400 <br />
** Dual Port DPDK <br />
*** 6 Rx @ 500 MS/s <br />
*** 8 Rx @ 400 MS/s <br />
*** 6 Tx @ 500 MS/s <br />
*** 8 Tx @ 450 MS/s <br />
*** 4 Rx + 4 Tx @ 500 MS/s <br />
*** 8 Rx + 8 Tx @ 250 MS/s <br />
* CG_1600 <br />
** Dual Port DPDK <br />
*** 2 Rx @ 1000 MS/s <br />
*** 2 Tx @ 1800 MS/s <br />
*** 2 Rx + 2 Tx @ 1000 MS/s <br />
<br />
All testing was done using dual 100GbE ports. For a single port, expect around half the number of channels as compared to the dual port equivalent configuration at the same streaming rate. <br />
<br />
==Guidance on SFP+ Adapters for Fiber Connectivity on NI Ettus USRP X4x0==<br />
<br />
Ettus Research currently offers direct-connect, copper cabling accessories for the NI Ettus USRP X4x0. However, it is also possible to use multi-mode fiber instead of copper connections for these devices. In this section, we will provide general guidance on the types of fiber adapters and cables that can be used with these products.<br />
<br />
The NI Ettus USRP X4x0 is compatible with most brands of SFP+ fiber adapters. In some cases, other equipment in the systems such as 1/10/100 Gigabit Ethernet switches are only compatible with specific brands of SFP+ adapters and cables. As a general rule, we recommend checking compatibility with the switches and network cards in your system before purchasing an adapter.<br />
<br />
Ettus Research does test the NI Ettus USRP X4x0 devices with the listed hardware as noted in the above section https://kb.ettus.com/X410#100_Gigabit_Ethernet<br />
<br />
==Certifications==<br />
===RoHS===<br />
As of December 1st, 2010 all Ettus Research products are RoHS compliant unless otherwise noted. More information can be found at [http://ettus.com/legal/rohs-information http://ettus.com/legal/rohs-information]<br />
<br />
===China RoHS=== <br />
'''Management Methods for Controlling Pollution Caused by Electronic Information Products Regulation'''<br />
<br />
'''Chinese Customers''' <br />
<br />
National Instruments is in compliance with the Chinese policy on the Restriction of Hazardous Substances (RoHS) used in Electronic Information Products. For more information about the National Instruments China RoHS compliance, visit [http://www.ni.com/environment/rohs_china ni.com/environment/rohs_china].<br />
<br />
==Certificate / Letter of Volatility==<br />
<br />
Certification can be found on the [https://www.ni.com/en/support/documentation/product-certifications.html NI Product Certifications lookup tool]<br />
[https://www.ni.com/docs/en-US/bundle/ni-ettus-usrp-x410-lov/resource/378157a.pdf X410 LOV]<br />
<br />
==Downloads==<br />
[http://files.ettus.com/manual/md_fpga.html FPGA Resources]<br />
<br />
[http://files.ettus.com/binaries/uhd_stable/ UHD Stable Binaries]<br />
<br />
[https://github.com/EttusResearch/uhd UHD Source Code on Github]<br />
<br />
<br />
[[Category:Hardware Resources]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Knowledge_Base&diff=5874Knowledge Base2023-09-22T15:34:46Z<p>MichaelDickens: /* Getting Started Guides */ join X410/X440</p>
<hr />
<div>Welcome to the Ettus Research Knowledge Base (KB). The KB is continuously being updated and expanded. If you have any suggestions, or do not find what you are looking for, then please [http://www.ettus.com/contact Contact Us].<br />
__NOTOC__<br />
<div class="row"><br />
<div class="col-1-3"><br />
== [[Getting Started Guides|<i class="fa fa-road"></i> Getting Started Guides]] ==<br />
<br />
'''Motherboards'''<br />
* [[B200/B210/B200mini/B205mini Getting Started Guides|B200/B210/B200mini/B205mini]]<br />
* [[Ettus USRP E300 Embedded Family Getting Started Guides|E310/E312/E313]]<br />
* [[E320 Getting Started Guide|E320]]<br />
* [[N200/N210 Getting Started Guides|N200/N210]]<br />
* [[USRP N300/N310/N320/N321 Getting Started Guide|N300/N310/N320/N321]]<br />
* [[X300/X310 Getting Started Guides|X300/X310]]<br />
* [[USRP-2974 Getting Started Guide|USRP-2974]]<br />
* [[USRP X410/X440 Getting Started Guide|X410/X440]]<br />
<br />
'''Daughterboards'''<br />
* [[BasicTX/BasicRX Getting Started Guides|BasicTX/BasicRX]]<br />
* [[CBX Getting Started Guides|CBX]]<br />
* [[LFTX/LFRX Getting Started Guides|LFTX/LFRX]]<br />
* [[SBX Getting Started Guides|SBX]]<br />
* [[TwinRX Getting Started Guides|TwinRX]]<br />
* [[UBX Getting Started Guides|UBX]]<br />
* [[WBX Getting Started Guides|WBX]]<br />
<br />
'''Other'''<br />
* [[Getting_Started_with_RFNoC_in_UHD_4.0|RFNoC Development (UHD 4.x)]]<br />
* [[RFNoC_4_Migration_Guide|RFNoC Migration Guide (UHD 3.x to UHD 4.x)]]<br />
* [[Getting_Started_with_RFNoC_Development|RFNoC Development (UHD 3.x)]]<br />
* [[Live SDR Environment Getting Started Guides|Live SDR Environment]]<br />
* [[OctoClock CDA-2990 Getting Started Guides|OctoClock CDA-2990]]<br />
* [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices|White Rabbit]]<br />
* [[Getting Started with DPDK and UHD|DPDK]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Hardware Resources|<i class="fa fa-cogs"></i> Hardware Resources]] ==<br />
'''Motherboards'''<br />
* [[B200/B210/B200mini/B205mini]]<br />
* [[Ettus USRP E300 Embedded Family Hardware Resources|E310/E312/E313]]<br />
* [[E320|E320]]<br />
* [[N200/N210]]<br />
* [[N300/N310]]<br />
* [[N320/N321]]<br />
* [[X300/X310]]<br />
* [[USRP-2974]]<br />
* [[X410]]<br />
* [[X440]]<br />
<br />
'''Daughterboards'''<br />
* [[BasicTX/BasicRX]]<br />
* [[CBX]]<br />
* [[LFTX/LFRX]]<br />
* [[SBX]]<br />
* [[TwinRX]]<br />
* [[UBX]]<br />
* [[WBX]]<br />
<br />
'''Other'''<br />
* [[OctoClock CDA-2990]]<br />
* [[GPSDO]]<br />
* [[Antennas]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Software Resources|<i class="fa fa-desktop"></i> Software Resources]] ==<br />
'''Ettus Products'''<br />
* [[UHD]]<br />
* [[UHD Python API]]<br />
* [[Getting_Started_with_RFNoC_in_UHD_4.0|RFNoC (UHD 4.x)]]<br />
* [[RFNoC]] (UHD 3.x)<br />
<br />
'''Third Party'''<br />
* [[GNU Radio]]<br />
* [[LabVIEW]]<br />
* [[Matlab/Simulink]]<br />
* [[OpenBTS]]<br />
* [[Eurecom OpenAirInterface (OAI)]]<br />
* [[srsLTE/srsUE]]<br />
* [[Gqrx]]<br />
* [[Fosphor]]<br />
<br />
'''Reference Architectures'''<br />
* [[Multichannel RF Reference Architecture]]<br />
<br />
</div><br />
</div><br />
<br />
<div class="row"><br />
<div class="col-1-3"><br />
<br />
== [[UHD and USRP User Manual|<i class="fa fa-flag"></i> UHD and USRP User Manual]] ==<br />
<br />
'''Software'''<br />
* [https://files.ettus.com/manual/ UHD Manual (master)]<br />
* [https://files.ettus.com/manual_archive/ UHD Manual Archive (previous releases)]<br />
<br />
'''Motherboards'''<br />
* [https://files.ettus.com/manual/page_usrp_b200.html B200/B210/B200mini/B205mini]<br />
* [https://files.ettus.com/manual/page_usrp_x3x0.html X300/X310]<br />
* [https://files.ettus.com/manual/page_usrp2.html N200/N210]<br />
* [https://files.ettus.com/manual/page_usrp_n3xx.html N300/N310/N320/N321]<br />
* [https://files.ettus.com/manual/page_usrp_e3xx.html E310/E312/E313/E320]<br />
* [https://files.ettus.com/manual/page_usrp_x4xx.html X410/X440]<br />
<br />
<br />
'''Daughterboards'''<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_basictx BasicRX/LFRX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_basicrx BasicTX/LFTX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_cbx CBX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_sbx SBX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_wbx WBX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_ubx UBX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_twinrx TwinRX]<br />
<br />
'''Other'''<br />
* [https://files.ettus.com/manual/page_octoclock.html OctoClock]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Application Notes|<i class="fa fa-file-text-o"></i> Application Notes]] ==<br />
Application Notes (AN) and technical articles written by engineers, for engineers. These articles offer experienced analysis, design ideas, reference designs, and tutorials—to make you productive and successful using USRP devices.<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Additional Resources|<i class="fa fa-book"></i> Additional Resources]] ==<br />
* [[Training]]<br />
* [[Suggested Reading|Suggested Reading]]<br />
* [[Suggested Videos|Suggested Videos]]<br />
* [[SDR Events]]<br />
* [[CGRAN]]<br />
</div><br />
</div><br />
<br />
<br />
<div class="row"><br />
<div class="col-1-3"><br />
<br />
== [[Technical Support|<i class="fa fa-life-ring"></i> Technical Support]] ==<br />
* [[Email|Email]]<br />
* [[Mailing Lists|Mailing Lists]]<br />
* [[Matrix|Matrix]]<br />
* [[Internet Relay Chat (IRC)|Internet Relay Chat (IRC)]]<br />
* [[StackExchange|StackExchange]]<br />
* [[Ordering and Fulfillment Help | Ordering and Fulfillment Help]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Faq|<i class="fa fa-info-circle"></i> FAQ]] ==<br />
* [[Technical FAQ|Technical]]<br />
* [[Licensing FAQ|Licensing]]<br />
<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Legacy Products| <i class="fa fa-hourglass-end"></i> Legacy Products]] ==<br />
'''Motherboards'''<br />
* [[USRP1|USRP1]]<br />
* [[USRP2|USRP2]]<br />
* [[E100/E110|E100/E110]]<br />
* [[B100]]<br />
<br />
'''Daughterboards'''<br />
* [[DBSRX2]]<br />
* [[TVRX2]]<br />
* [[XCVR2450]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=USRP_X410/X440_Getting_Started_Guide&diff=5873USRP X410/X440 Getting Started Guide2023-09-22T15:34:38Z<p>MichaelDickens: initial creation</p>
<hr />
<div>==Kit Contents==<br />
===X4x0===<br />
{|<br />
|style="vertical-align:top"|<br />
* NI Ettus USRP X410 or X440<br />
* DC Power Supply (12V, 20A)<br />
* 1 Gigabit Ethernet Cat-5e Cable (3m)<br />
* USB-A to USB-C Cable (1m)<br />
* Getting Started Guide URL (QR Code)<br />
* Safety, Environmental, and Regulatory Information<br />
||[[File:X410.jpg|450px|center]]<br />
||[[File:X440.jpg|450px|center]]<br />
|}<br />
<br />
==USRP X440 Design Considerations==<br />
* https://kb.ettus.com/About_Sampling_Rates_and_Master_Clock_Rates_for_the_USRP_X440<br />
<br />
==You Will Need==<br />
* For Network Mode: A host computer with an available 1 or 10 Gigabit Ethernet interface for sample streaming. In addition to the Ethernet interface used for sampling streaming, your host computer will require a separate 1 Gigabit Ethernet interface for command and control streaming.<br />
<br />
* For Stand-Alone Embedded Mode: A host computer with an available 1 Gigabit Ethernet port or a USB 2.0 port to remotely access the embedded Linux operating system running on ARM CPU.<br />
<br />
==Proper Care and Handling==<br />
<br />
All Ettus Research products are individually tested before shipment. The USRP is guaranteed to be functional at the time it is received by the customer. Improper use or handling of the USRP can cause the device to become non-functional. Take the following precautions to prevent damage to the unit.<br />
<br />
* Never allow metal objects to touch the circuit board while powered.<br />
* Always properly terminate the transmit port with an antenna or 50Ω load.<br />
* Always handle the board with proper anti-static methods.<br />
* Never allow the board to directly or indirectly come into contact with any voltage spikes.<br />
* Never allow any water or condensing moisture to come into contact with the device.<br />
* Always use caution with FPGA, firmware, or software modifications.<br />
<br />
{|<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |X410: Never apply more than +14 dBm continuous <=3GHz, +17 dBm continuous >3GHz, or +20dBm more than 5 minutes >3GHz of power into any RF input.<br />
|-<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |X440: Never apply more than +13 dBm continuous <=2.5GHz, +17 dBm continuous between 2.5GHz and 3.6 GHz, or +20dBm continuous between 3.6 GHz and 4 GHz of power into any RF input.<br />
|-<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |[[File:Caution.png|24px|center]]<br />
|style="padding-left:10px; padding-right:10px; padding-bottom:10px;" |X440: Always use at least 30dB attenuation if operating in loopback configuration<br />
|-<br />
|}<br />
<br />
==Install and Setup the Software Tools on Your Host Computer==<br />
In order to use your Universal Software Radio Peripheral (USRP™), you must have the software tools correctly installed and configured on your host computer. The easiest way to install USRP Hardware Driver (UHD) is by getting a binary installer package for your operating system as described in the UHD manual about [https://files.ettus.com/manual/page_install.html Binary Installation]. If no binary packages are available for your operating system or you want to modify the sources by yourself, a step-by-step guide is available at the Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux|Linux]], [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X|OS X]] and [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows|Windows]] Application Notes.<br />
<br />
To find the latest release of UHD, see the UHD repository at https://github.com/EttusResearch/uhd.<br />
<br />
The USRP X410 requires UHD version 4.1 or later.<br />
The USRP X440 requires UHD version 4.5 or later. <br />
<br />
'''When you receive a brand-new device, it is strongly recommended that you download the latest filesystem image from the Ettus Research website update the unit. It is not recommended that you use the filesystem from the factory as-is. Instructions on downloading the latest filesystem image and updating it is listed below.'''<br />
<br />
'''Note that if you are operating the device in Network Mode, the version of UHD running on the host computer and the USRP X4x0 must match.'''<br />
<br />
==Assembling the X4x0==<br />
Inside the kit you will find the X4x0 and an X4x0 power supply. Plug these in, connect the 1GbE RJ45 interface to your network, and power on the device by pressing the power button.<br />
<br />
<br />
==The STM32 Microcontroller==<br />
<br />
The STM32 microcontroller (also referred to as the "SCU") controls various low-level features of the X4x0 series motherboard: It controls the power sequencing, reads out fan speeds and some of the temperature sensors. It is connected to the RFSoC via an I2C bus. It is running software based on Chromium EC.<br />
<br />
It is possible to log into the STM32 using the serial interface (see Connecting to the Microcontroller). This will allow certain low-level controls, such as remote power cycling should the CPU have become unresponsive for whatever reason.<br />
<br />
===Updating the SCU===<br />
<br />
The writable SCU image file is stored on the filesystem under /lib/firmware/ni/ec-titanium-revX.RW.bin (where X is a revision compatibility number). To update, simply replace the .bin file with the updated version and reboot.<br />
<br />
==eMMC Storage==<br />
<br />
The main non-volatile storage of the USRP is a 16 GB eMMC storage. This storage can be made accessible as a USB Mass Storage device through the USB-OTG connector on the back panel.<br />
<br />
The entire root file system (Linux kernel, libraries) and any user data are stored on the eMMC. It is partitioned into four partitions:<br />
<br />
Boot partition (contains the bootloader). This partition usually does not require modification.<br />
A data partition, mounted in /data. This is the only partition that is not erased during file system updates.<br />
Two identical system partitions (root file systems). These contain the operating system and the home directory (anything mounted under / that is not the data or boot partition). The reason there are two of these is to enable remote updates: An update running on one partition can update the other one without any effect to the currently running system. Note that the system partitions are erased during updates and are thus unsuitable for permanently storing information.<br />
Note: It is possible to access the currently inactive root file system by mounting it. After logging into the device using serial console or SSH (see the following two sections), run the following commands:<br />
<br />
<pre><br />
$ mkdir temp<br />
<br />
$ mount /dev/mmcblk0p3 temp # This assumes mmcblk0p3 is currently not mounted<br />
<br />
$ ls temp # You are now accessing the idle partition:<br />
<br />
bin data etc lib media proc sbin tmp usr<br />
boot dev home lost+found mnt run sys uboot var<br />
</pre><br />
<br />
The device node in the mount command might differ, depending on which partition is currently already mounted.<br />
<br />
==USB Access to eMMC==<br />
<br />
While Mender should be used for routine filesystem updates (see Updating Filesystems), it is also possible to access the X4x0's internal eMMC from an external host over USB. This allows accessing or modifying the filesystem, as well as the ability to flash the device with an entirely new filesystem.<br />
<br />
In order to do so, you'll need an external computer with two USB ports, and two USB cables to connect the computer to your X4x0. The instructions below assume a Linux host.<br />
<br />
First, connect to the APU serial console at a baud rate of 115200. Boot the device, and stop the boot sequence by typing noautoboot at the prompt. Then, run the following command in the U-boot command prompt:<br />
<br />
<code>ums 0 mmc 0</code><br />
<br />
This will start the USB mass storage gadget to expose the eMMC as a USB mass storage device. You should see a spinning indicator on the console, which indicates the gadget is active.<br />
<br />
Next, connect your external computer to the X4x0's USB to PS port using an OTG cable. Your computer should recognize the X4x0 as a mass storage device, and you should see an entry in your kernel logs (dmesg) that looks like this:<br />
<br />
<pre><br />
usb 3-1: New USB device found, idVendor=3923, idProduct=7a7d, bcdDevice= 2.23<br />
usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0<br />
usb 3-1: Product: USB download gadget<br />
usb 3-1: Manufacturer: National Instruments<br />
sd 6:0:0:0: [sdc] 30932992 512-byte logical blocks: (15.8 GB/14.8 GiB)<br />
sdc: sdc1 sdc2 sdc3 sdc4<br />
sd 6:0:0:0: [sdc] Attached SCSI removable disk<br />
</pre><br />
<br />
The exact output will depend on your machine, but from this log you can see that the X4x0 was recognized and /dev/sdc is the block device representing the eMMC, with 4 partitions detected (see eMMC Storage for details on the partition layout).<br />
<br />
It is now possible to treat the X4x0's eMMC as you would any other USB drive: the individual partitions can be mounted and accessed, or the entire block device can be read/written.<br />
<br />
Once you're finished accessing the device over USB, the u-boot gadget may be stopped by hitting Ctrl-C at the APU serial console.<br />
<br />
<br />
== Flashing the eMMC ==<br />
<br />
Once the X4x0's eMMC is accessible over USB, it's possible to write the filesystem image and thus change the device's filesystem. You can obtain the latest filesystem image by running:<br />
<br />
<code>uhd_images_downloader -t sdimg -t x4xx</code><br />
<br />
The output of this command will indicate where the downloaded images were put, or specify a custom location using using the <code>-i INSTALL_LOCATION</code> argument.<br />
<br />
There are 2 ways to write the image to the X4x0's eMMC: using <code>dd</code> and <code>bmaptool</code>. Run one of the following commands, replacing <code>/dev/sdX</code> with the block device of the X4x0's eMMC (found in the device's kernel log or by running <code>lsblk</code>). Take care to use the correct block device or else you might overwrite the wrong drive!<br />
<br />
<code>sudo dd if=/path/to/usrp_x4xx_fs.sdimg of=/dev/sdX bs=1M</code><br />
<br />
<code>sudo bmaptool copy --bmap /path/to/usrp_x4xx_fs.sdimg.bmap /path/to/usrp_x4xx_fs.sdimg /dev/sdX</code><br />
<br />
The former is generally preferred as it will always work, even if it slower than the latter.<br />
<br />
==Using a USRP X4x0 from UHD==<br />
Like any other USRP, all X4x0 USRPs are controlled by the UHD software. To integrate a USRP X4x0 into your C++ application, you would generate a UHD device in the same way you would for any other USRP:<br />
<br />
<code>auto usrp = uhd::usrp::multi_usrp::make("type=x4xx");</code><br />
<br />
For a list of which arguments can be passed into make(), see Section Device Arguments.<br />
<br />
==Updating Filesystems==<br />
<br />
Mender is a third-party software that enables remote updating of the root file system without physically accessing the device (see also the [https://mender.io/ Mender website]). Mender can be executed locally on the device, or a Mender server can be set up which can be used to remotely update an arbitrary number of USRP devices. Mender servers can be self-hosted, or hosted by Mender (see mender.io for pricing and availability).<br />
<br />
When updating the file system using Mender, the tool will overwrite the root file system partition that is not currently mounted (note: the onboard flash storage contains two separate root file system partitions, only one is ever used at a single time). Any data stored on that partition will be permanently lost, including the currently loaded FPGA image. After updating that partition, it will reboot into the newly updated partition. Only if the update is confirmed by the user, the update will be made permanent. This means that if an update fails, the device will be always able to reboot into the partition from which the update was originally launched (which presumably is in a working state). Another update can be launched now to correct the previous, failed update, until it works.<br />
<br />
To obtain the file system Mender image (these are files with a <code>.mender</code> suffix), run the following command on the host computer with Internet access:<br />
<br />
$ sudo uhd_images_downloader -t mender -t x4xx --yes<br />
<br />
NOTE: In the output of the command, the folder destination where the images are saved is printed out.<br />
<br />
Next, you will need to copy this Mender file system image to the USRP X4xx. This can be done with the Linux utility <code>scp</code>.<br />
<br />
$ scp /usr/local/share/uhd/images/usrp_x4xx_fs.mender root@192.168.1.51:~/. <br />
<br />
Note: The path and IP may different for your configuration, the command above assumes you're using the default installation path of <code>/usr/local</code> and that the X4xx's IP is <code>192.168.1.51</code>.<br />
<br />
After copying the Mender file system image to the X4xx, connect to the X4xx using either the Serial Console, or via SSH to gain shell access.<br />
<br />
On the X4xx, run <code>mender install /path/to/latest.mender</code> to update the file system:<br />
<br />
$ mender install /home/root/usrp_x4xx_fs.mender<br />
<br />
The artifact can also be stored on a remote server:<br />
$ mender install <nowiki>http://server.name/path/to/latest.mender</nowiki><br />
<br />
This procedure will take a few minutes to complete. After mender has logged a successful update, reboot the device:<br />
$ reboot<br />
<br />
If the reboot worked, and the device seems functional, commit the changes so that the boot loader knows to permanently boot into this partition:<br />
$ mender -commit<br />
<br />
To identify the currently installed Mender artifact from the command line, the following file can be queried on the X4x0:<br />
$ cat /etc/mender/artifact_info<br />
<br />
If you are using a Mender server, the updates can be initiated from a web dashboard. From there, you can start the updates without having to log into the device, and you can update groups of USRPs with a few clicks in a web GUI. The dashboard can also be used to inspect the state of USRPs. This is a simple way to update groups of rack-mounted USRPs with custom file systems.<br />
<br />
If you are running a hosted server, the updates can be initiated from a web dashboard. From there, you can start the updates without having to log into the device, and can update groups of USRPs with a few clicks in a web GUI. The dashboard can also be used to inspect the state of USRPs. This is a simple way to update groups of rack-mounted USRPs with custom file systems.<br />
<br />
==Network Interfaces==<br />
The Ettus USRP X4x0 has various network interfaces:<br />
<br />
eth0: RJ45 port.<br />
<br />
The RJ45 port comes up with a default configuration of DHCP, that will request a network address from your DHCP server (if available on your network). This interface is agnostic of FPGA image flavor.<br />
<br />
int0: internal interface for network communication between the embedded ARM processor and FPGA.<br />
<br />
The internal network interface is configured with a static address: 169.254.0.1/24. This interface is agnostic of FPGA image flavor.<br />
<br />
sfpX [, sfpX_1, sfpX_2, sfpX_3]: QSFP28 network interface(s), up-to four (one per lane) based on implemented protocol.<br />
<br />
Each QSFP28 port has four high-speed transceiver lanes. Therefore, depending on the FPGA image flavor, up-to four different network interfaces may exist per QSFP28 port, using the sfpXfor the first lane, and sfpX_1-3 for the other three lanes. Each network interface has a default static IP address. Note that for multi-lane protocols, such as 100 GbE, a single interface is used (sfpX).<br />
The configuration files for these network interfaces are stored in: <code>/data/network/</code><br />
<br />
{| class="wikitable" <br />
|-<br />
! Interface Name<br />
! Description<br />
! Default Configuration<br />
! Configuration File<br />
! Example: X4_200/X4_400 FPGA image<br />
|-<br />
| eth0<br />
| RJ45<br />
| style="vertical-align:middle; background-color:#FFF;" | DHCP<br />
| style="vertical-align:middle; background-color:#FFF;" | eth0.network<br />
| DHCP<br />
|-<br />
| int0<br />
| Internal<br />
| style="vertical-align:middle; background-color:#FFF;" | 169.254.0.1/24<br />
| style="vertical-align:middle; background-color:#FFF;" | int0.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 169.254.0.1/24<br />
|-<br />
| sfp0<br />
| QSFP28 0 (4-lanes interface or lane 0)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.10.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp0.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.10.2/24<br />
|-<br />
| style="background-color:#FFF;" | sfp0_1<br />
| QSFP28 0 (lane 1)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.11.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp0_1.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.11.2/24<br />
|-<br />
| style="background-color:#FFF;" | sfp0_2<br />
| QSFP28 0 (lane 2)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.12.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp0_2.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.12.2/24<br />
|-<br />
| sfp0_3<br />
| QSFP28 0 (lane 3)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.13.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp0_3.network<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.13.2/24<br />
|-<br />
| sfp1<br />
| QSFP28 1 (4-lanes interface or lane 0)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.20.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp1.network<br />
| style="vertical-align:middle; background-color:#FFF;" | N/C<br />
|-<br />
| sfp1_1<br />
| QSFP28 1 (lane 1)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.21.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp1_1.network<br />
| style="vertical-align:middle; background-color:#FFF;" | N/C<br />
|-<br />
| style="background-color:#FFF;" | sfp1_2<br />
| QSFP28 1 (lane 2)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.22.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp1_2.network<br />
| style="vertical-align:middle; background-color:#FFF;" | N/C<br />
|-<br />
| style="background-color:#FFF;" | sfp1_3<br />
| QSFP28 1 (lane 3)<br />
| style="vertical-align:middle; background-color:#FFF;" | 192.168.23.2/24<br />
| style="vertical-align:middle; background-color:#FFF;" | sfp1_3.network<br />
| style="vertical-align:middle; background-color:#FFF;" | N/C<br />
|-<br />
|}<br />
<br />
==Network Connectivity==<br />
Once the X4x0 has booted, determine the IP address and verify network connectivity by running uhd_find_devices on the host computer:<br />
<br />
X410:<br />
<pre><br />
$ uhd_find_devices<br />
<br />
-- UHD Device 0<br />
<br />
Device Address:<br />
serial: 1234ABC<br />
addr: 10.2.161.10<br />
claimed: False<br />
mgmt_addr: 10.2.161.10<br />
product: x410<br />
type: x4xx<br />
</pre><br />
<br />
X440:<br />
<pre><br />
$ uhd_find_devices<br />
<br />
-- UHD Device 0<br />
<br />
Device Address:<br />
serial: 1234ABC<br />
addr: 10.2.161.10<br />
claimed: False<br />
mgmt_addr: 10.2.161.10<br />
product: x440<br />
type: x4xx<br />
</pre><br />
<br />
By default, an X4x0 will use DHCP to attempt to find an address.<br />
<br />
At this point, you should run:<br />
<br />
<code>uhd_usrp_probe --args addr=<IP address></code><br />
to ensure functionality of the device.<br />
<br />
Note: If you receive the following error:<br />
<br />
<code>Error: RuntimeError: Graph edge list is empty for rx channel 0</code><br />
then you will need to download a UHD-compatible FPGA as described in Updating the FPGA or using the following command (it assumes that FPGA images have been downloaded previously using uhd_images_downloader, or that the command is run on the device itself):<br />
<br />
X410:<br />
<code>uhd_image_loader --args type=x4xx,addr=<ip address>,fpga=X4_200</code><br />
<br />
X440:<br />
<code>uhd_image_loader --args type=x4xx,addr=<ip address>,fpga=X4_400</code><br />
<br />
When running on the device, use <code>127.0.0.1</code> as the IP address.<br />
<br />
You can now use existing UHD examples or applications (such as rx_sample_to_file, rx_ascii_art_dft, or tx_waveforms) or other UHD-compatible applications to start receiving and transmitting with the device.<br />
<br />
See Network Interfaces for further details on the various network interfaces available on the X4x0.<br />
<br />
<br />
===Network Status LEDs===<br />
The Ettus USRP X4x0 is equipped with status LEDs for its network-capable ports: RJ45 and QSFP28s, see RJ45 LED Behavior and QSFP28 LED Behavior accordingly.<br />
<br />
====RJ45 LED Behavior====<br />
The RJ45 port has two independent LEDs: green (right) and yellow (left). The table below summarizes the LEDs' behavior. Note that link speed indication is not currently supported.<br />
<br />
{| class="wikitable" <br />
|- style="font-weight:bold; text-align:center; vertical-align:middle;"<br />
! Link / Activity<br />
! Green LED<br />
! Yellow LED<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | No Link<br />
| Off<br />
| Off<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | Link / No Activity<br />
| On<br />
| Off<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | Link / Activity<br />
| On<br />
| Blinking<br />
|}<br />
<br />
====QSFP28 LED Behavior====<br />
Each QSFP28 connector has four LEDs, one for each high-speed transceiver lane. The table below summarizes the LEDs' behavior, note that for multi-lane protocols, such as 100 GbE, the corresponding LEDs are ganged together. Within the same image, multiple speeds on the same port (e.g., both 10 GbE and 100 GbE) are not supported, therefore link speed indication is not supported.<br />
<br />
{| class="wikitable" <br />
|- style="font-weight:bold; text-align:center; vertical-align:middle;"<br />
! Link / Activity<br />
! QSFP28 LED (4 Total)<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | No Link<br />
| Off<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | Link / No Activity<br />
| Green (solid)<br />
|-<br />
| style="vertical-align:middle; background-color:#FFF;" | Link / Activity<br />
| Amber (blinking)<br />
|}<br />
<br />
==Security-related Settings==<br />
The X4x0 ships without a root password set. It is possible to ssh into the device by simply connecting as root, and thus gaining access to all subsystems. To set a password, run the command<br />
<br />
<code>$ passwd</code><br />
on the device.<br />
<br />
==Serial Connection==<br />
It is possible to gain access to the device using a serial terminal emulator. To do so, the USB debug port needs to be connected to a separate computer to gain access. Most Linux, OSX, or other Unix flavors have a tool called 'screen' which can be used for this purpose, by running the following command:<br />
<br />
<code>$ sudo screen /dev/ttyUSB2 115200</code><br />
In this command, we prepend 'sudo' to elevate user privileges (by default, accessing serial ports is not available to regular users), we specify the device node (in this case, /dev/ttyUSB2), and the baud rate (115200).<br />
<br />
The exact device node depends on your operating system's driver and other USB devices that might be already connected. Modern Linux systems offer alternatives to simply trying device nodes; instead, the OS might have a directory of symlinks under /dev/serial/by-id:<br />
<br />
<pre>$ ls /dev/serial/by-id<br />
usb-Digilent_Digilent_USB_Device_2516351DDCC0-if02-port0<br />
usb-Digilent_Digilent_USB_Device_2516351DDCC0-if03-port0<br />
</pre><br />
<br />
Note: Exact names depend on the host operating system version and may differ.<br />
<br />
The first (with the if02 suffix) connects to the STM32 microcontroller (SCU), whereas the second (with the if03 suffix) connects to Linux running on the RFSoC APU.<br />
<br />
<code>$ sudo screen /dev/serial/by-id/usb-Digilent_Digilent_USB_Device_2516351DDCC0-if03-port0 115200</code><br />
After entering the username root (no password is set by default), you should be presented with a shell prompt similar to the following:<br />
<br />
<code>root@ni-x4xx-1234ABC:~#</code><br />
On this prompt, you can enter any Linux command available. Using the default configuration, the serial console will also show all kernel log messages (unlike when using SSH, for example), and give access to the boot loader (U-boot prompt). This can be used to debug kernel or bootloader issues more efficiently than when logged in via SSH.<br />
<br />
==Connecting to the Microcontroller==<br />
The microcontroller (which controls the power sequencing, among other things) also has a serial console available. To connect to the microcontroller, use the other UART device. In the example above:<br />
<br />
<code>$ sudo screen /dev/serial/by-id/usb-Digilent_Digilent_USB_Device_2516351DDCC0-if02-port0 115200</code><br />
<br />
It provides a very simple prompt. The command 'help' will list all available commands. A direct connection to the microcontroller can be used to hard-reset the device without physically accessing it and other low-level diagnostics. For example, running the command reboot will emulate a reset button press, resetting the state of the device, while the command powerbtn will emulate a power button press, turning the device back on again.<br />
<br />
==SSH Connection==<br />
The USRP X4x0 has two network connections: The dual QSFP28 ports, and an RJ45 connector. The latter is by default configured by DHCP; by plugging it into into 1 Gigabit switch on a DHCP-capable network, it will get assigned an IP address and thus be accessible via ssh.<br />
<br />
In case your network setup does not include a DHCP server, refer to the section Serial Connection. A serial login can be used to assign an IP address manually.<br />
<br />
After the device obtained an IP address you can log in from a Linux or OSX machine by typing:<br />
<br />
<code>$ ssh root@ni-x4xx-1234ABC # Replace with your actual device name!</code><br />
Depending on your network setup, using a .local domain may work:<br />
<br />
<code>$ ssh root@ni-x4xx-1234ABC.local</code><br />
Of course, you can also connect to the IP address directly if you know it (or set it manually using the serial console).<br />
<br />
Note: The device's hostname is derived from its serial number by default (<code>ni-x4xx-$SERIAL</code>). You can change the hostname by creating the file <code>/data/network/hostname</code>, saving the desired hostname in it, then rebooting.<br />
<br />
On Microsoft Windows, the connection can be established using a tool such as PuTTY, by selecting a username of root without password.<br />
<br />
Like with the serial console, you should be presented with a prompt like the following:<br />
<br />
<code>root@ni-x4xx-1234ABC:~#</code><br />
<br />
== Autoboot ==<br />
<br />
The USRP X4x0 can be configured to power on and boot automatically when power is applied. This setting can be controlled using the <code>eeprom-set-autoboot</code> script. This script is executed directly on the USRP X4x0. To enable autoboot, run <code>eeprom-set-autoboot on</code>; to disable autoboot, run <code>eeprom-set-autoboot off</code>.<br />
<br />
==Updating the FPGA==<br />
<br />
The FPGA can be updated simply using uhd_image_loader:<br />
<br />
<code>uhd_image_loader --args type=x4xx,addr=<IP address of device> --fpga-path <path to .bit></code><br />
or<br />
<br />
<code>uhd_image_loader --args type=x4xx,addr=<IP address of device>,fpga=FPGA_TYPE</code><br />
A UHD install will likely have pre-built images in /usr/share/uhd/images/. Up-to-date images can be downloaded using the uhd_images_downloader script:<br />
<br />
<code>uhd_images_downloader</code><br />
will download images into /usr/share/uhd/images/ (the path may differ, depending on how UHD was installed).<br />
<br />
Also note that the USRP already ships with compatible FPGA images on the device - these images can be loaded by SSH'ing into the device and running:<br />
<br />
X410:<br />
<code>uhd_image_loader --args type=x4xx,mgmt_addr=127.0.0.1,fpga=X4_200</code><br />
<br />
X440:<br />
<code>uhd_image_loader --args type=x4xx,mgmt_addr=127.0.0.1,fpga=X4_400</code><br />
<br />
==FPGA Image Flavors==<br />
Unlike the USRP X310 or other third-generation USRP devices, the FPGA image flavors do not only encode how the QSFP28 connectors are configured, but also which master clock rates are available. This is because the data converter configuration is part of the FPGA image (the ADCs/DACs on the X4x0 are on the same die as the FPGA). The image flavors consist of two short strings, separated by an underscore, e.g. X4_200 (X410) or X4_400 (X440) is an image flavor which contains 4x 10 GbE, and can handle an analog bandwidth of 200 MHz or 400 MHz respectively. The first two characters describe the configuration of the QSFP28 ports: 'X' stands for 10 GbE, 'C' stands for 100 GbE. For details see[https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_updating_fpga_types FPGA Image Flavor] in the [https://files.ettus.com/manual USRP Hardware Driver and USRP Manual].<br />
<br />
<br />
The analog bandwidth determines the available master clock rates. <br />
<br />
X410: As of UHD 4.1, only the X4_200 image is shipped with UHD, which allows a 245.76 MHz or 250 MHz master clock rate. With UHD 4.2, the CG_400 image was added allowing for 491.52 MHz and 500 MHz master clock rates. With UHD 4.5, the UC_200 image (245.76 MHz and 250 MHz master clock rate) was added.<br />
<br />
<br />
X440: As of UHD 4.5, UHD ships with X4_400, X4_1600, CG_400 and CG_1600 images. The X4_400 and CG_400 images allow master clock rates between 125 MHz and 512 MHz and the usage of all 8 channels while the X4_1600 and CG_1600 images allow master clock rates between 125 MHz and 2048 MHz but only the usage of channels 0 and 4.<br />
<br />
Any other images are considered experimental (unsupported).<br />
<br />
==Device Arguments==<br />
<br />
{| class="wikitable" style="vertical-align:middle;"<br />
|- style="font-weight:bold; text-align:center;"<br />
! Key<br />
! Description<br />
! Example Value<br />
|-<br />
| addr<br />
| IPv4 address of primary SFP+ port to connect to.<br />
| addr=192.168.30.2<br />
|-<br />
| second_addr<br />
| IPv4 address of secondary SFP+ port to connect to.<br />
| second_addr=192.168.40.2<br />
|-<br />
| mgmt_addr<br />
| IPv4 address or hostname to which to connect the RPC client. Defaults to `addr'.<br />
| mgmt_addr=ni-sulfur-311FE00<br />
|-<br />
| find_all<br />
| When using broadcast, find all devices, even if unreachable via CHDR.<br />
| find_all=1<br />
|-<br />
| master_clock_rate<br />
| Master Clock Rate in Hz.<br />
| master_clock_rate=250e6<br />
|-<br />
| converter_rate<br />
| Converter Rate in Hz. Only X440 and together with master_clock_rate.<br />
| master_clock_rate=250e6,converter_rate=1000e6<br />
|-<br />
| serialize_init<br />
| Force serial initialization of daughterboards.<br />
| serialize_init=1<br />
|-<br />
| skip_init<br />
| Skip the initialization process for the device.<br />
| skip_init=1<br />
|-<br />
| time_source<br />
| Specify the time (PPS) source.<br />
| time_source=internal<br />
|-<br />
| clock_source<br />
| Specify the reference clock source.<br />
| clock_source=internal<br />
|-<br />
| ref_clk_freq<br />
| Specify the external reference clock frequency, default is 10 MHz.<br />
| ref_clk_freq=20e6<br />
|-<br />
| discovery_port<br />
| Override default value for MPM discovery port.<br />
| discovery_port=49700<br />
|-<br />
| rpc_port<br />
| Override default value for MPM RPC port.<br />
| rpc_port=49701<br />
|}<br />
<br />
This is only a subset of the existing device arguments. For a complete list please consult the [https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_args UHD user manual of the X4x0 device series]. <br />
<br />
==GPS==<br />
<br />
The USRP X4x0 includes a Jackson Labs LTE-Lite GPS module. Its antenna port is on the rear panel (see Front and Back Panels). When the X4x0 has access to GPS satellite signals, it can use this module to read out the current GPS time and location as well as to discipline an onboard OCXO.<br />
<br />
To use the GPS as a clock and time reference, simply use gpsdo as a clock or time source. Alternatively, set gpsdo as a synchronization source:<br />
<br />
<pre><br />
// Set clock/time individually:<br />
usrp->set_clock_source("gpsdo");<br />
usrp->set_time_source("gpsdo");<br />
// This is equivalent to the previous commands, but faster, as it sets<br />
// both settings simultaneously and avoids duplicating settings that are shared<br />
// between these calls.<br />
usrp->set_sync_source("clock_source=gpsdo,time_source=gpsdo");<br />
</pre><br />
<br />
Note the GPS module is not always enabled. Its power-on status can be queried using the gps_enabled GPS sensor (see also The Sensor API). When disabled, none of the sensors will return useful (if any) values.<br />
<br />
When selecting gpsdo as a clock source, the GPS will always be enabled. Note that acquiring a GPS lock can take some time after enabling the GPS, so if a UHD application is enabling the GPS dynamically, it might take some time before a GPS lock is reported.<br />
<br />
==Front-Panel Programmable GPIOs==<br />
<br />
The USRP X4x0 has two HDMI front-panel connectors, which are connected to the FPGA. For a <br />
description of the GPIO control API, see the<br />
[https://files.ettus.com/manual/page_x400_gpio_api.html USRP X4x0 GPIO UHD Manual Entry],<br />
[https://files.ettus.com/manual/page_usrp_x4xx.html#x4xx_usage_gpio the USRP X4x0 Series Manual],<br />
the [https://files.ettus.com/manual/page_zbx.html#zbx_atr ZBX ATR section] (X410) and the<br />
[https://files.ettus.com/manual/page_fbx.html#fbx_atr FBX ATR section] (X440).<br />
<br />
<br />
==Subdev Specifications==<br />
<br />
The RF ports on the front panel of the X410 + ZBX correspond to the following subdev specifications:<br />
<br />
{| class="wikitable" <br />
|-<br />
! Label<br />
! style="text-align:center; vertical-align:middle; font-weight:bold;" | Subdev Spec<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 0<br />
| A:0<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 1<br />
| A:1<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 0<br />
| B:0<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 1<br />
| B:1<br />
|}<br />
<br />
The RF ports on the front panel of the X440 + FBX correspond to the following subdev specifications (for xx_400 FPGA images):<br />
<br />
{| class="wikitable" <br />
|-<br />
! Label<br />
! style="text-align:center; vertical-align:middle; font-weight:bold;" | Subdev Spec<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 0<br />
| A:0<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 1<br />
| A:1<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 2<br />
| A:2<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 0 / RF 3<br />
| A:3<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 0<br />
| B:0<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 1<br />
| B:1<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 2<br />
| B:2<br />
|- style="vertical-align:middle; background-color:#FFF;"<br />
| DB 1 / RF 3<br />
| B:3<br />
|}<br />
<br />
When using a xx_1600 FPGA image on X440, only A:0 and B:0 are available.<br />
<br />
The subdev spec slot identifiers "A" and "B" are not reflected on the front panel. They were set to match valid subdev specifications of previous USRPs, maintaining backward compatibility.<br />
<br />
These values can be used for uhd::usrp::multi_usrp::set_rx_subdev_spec() and uhd::usrp::multi_usrp::set_tx_subdev_spec() as with other USRPs.<br />
<br />
<br />
==Rear Panel Status LEDs==<br />
<br />
The USRP X4x0 is equipped with four LEDs located on the device's rear panel. Each LED supports four different states: Off, Green, Red, and Amber. One LED (PWR) indicates the device's power state (see Power LED below). The other three LEDs (LED 0, LED 1, and LED 2) are user-configurable, different behaviors are supported for each of these LEDs (see User-configurable LEDs below).<br />
<br />
[[File:x4xx_rearpanel_status_leds.png|125px]]<br />
<br />
===X4x0 Rear Panel Status LEDs===<br />
Power LED<br />
The USRP X4x0's PWR LED is reserved to visually indicate the user the device's power state. Power LED Behavior describes what each LED state represents.<br />
<br />
===Power LED Behavior===<br />
<br />
{| class="wikitable" style="background-color:#FFF;"<br />
|- style="font-weight:bold; text-align:center;"<br />
! PWR LED State<br />
! style="vertical-align:middle;" | Meaning<br />
|- style="vertical-align:middle;"<br />
| Off<br />
| No power is applied<br />
|- style="vertical-align:middle;"<br />
| Amber<br />
| Power is good but X4x0 is powered off<br />
|- style="vertical-align:middle;"<br />
| Green<br />
| Power is good and X4x0 is powered on<br />
|- style="vertical-align:middle;"<br />
| Red<br />
| Power error state<br />
|}<br />
<br />
===User-configurable LEDs===<br />
The USRP X4x0's user-configurable rear panel status LEDs (LED 0, LED 1, and LED 2) allow the user to have visual indication of various device conditions. Supported LED Behaviors provides a complete list of the supported behaviors for each user-configurable LED. By default, these LEDs are configured as described in LEDs Default Behavior.<br />
<br />
The user may alter the default LEDs behavior either temporarily or persistently, see the Temporarily change the LED Behavior or Persistently in the UHD manual to change the LED Behavior accordingly.<br />
<br />
https://files.ettus.com/manual/page_usrp_x4xx.html<br />
<br />
==Technical Support and Community Knowledge Base==<br />
Technical support for USRP hardware is available through email only. If the product arrived in a nonfunctional state or you require technical assistance, please contact [mailto:support@ettus.com support@ettus.com]. Please allow 24 to 48 hours for response by email, depending on holidays and weekends, although we are often able to reply more quickly than that.<br />
<br />
We also recommend that you subscribe to the community mailing lists. The mailing lists have a responsive and knowledgeable community of hundreds of developers and technical users who are located around the world. When you join the community, you will be connected to this group of people who can help you learn about SDR and respond to your technical and specific questions. Often your question can be answered quickly on the mailing lists. Each mailing list also provides an archive of all past conversations and discussions going back many years. Your question or problem may have already been addressed before, and a relevant or helpful solution may already exist in the archive.<br />
<br />
Discussions involving the USRP hardware and the UHD software itself are best addressed through the '''usrp-users''' mailing list at [http://usrp-users.ettus.com http://usrp-users.ettus.com].<br />
<br />
Discussions involving the use of [http://gnuradio.org/ GNU Radio] with USRP hardware and UHD software are best addressed through the '''discuss-gnuradio''' mailing list at [https://lists.gnu.org/mailman/listinfo/discussgnuradio https://lists.gnu.org/mailman/listinfo/discussgnuradio].<br />
<br />
Discussions involving the use of [http://openbts.org/ OpenBTS®] with USRP hardware and UHD software are best addressed through the '''openbts-discuss''' mailing list at [https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss https://lists.sourceforge.net/lists/listinfo/openbtsdiscuss].<br />
<br />
The support page on our website is located at [https://www.ettus.com/support https://www.ettus.com/support]. The Knowledge Base is located at [https://kb.ettus.com https://kb.ettus.com].<br />
<br />
==Legal Considerations==<br />
Every country has laws governing the transmission and reception of radio signals. Users are solely responsible for insuring they use their USRP system in compliance with all applicable laws and regulations. Before attempting to transmit and/or receive on any frequency, we recommend that you determine what licenses may be required and what restrictions may apply.<br />
<br />
*NOTE: This USRP product is a piece of test equipment.<br />
<br />
==Sales and Ordering Support==<br />
If you have any non-technical questions related to your order, then please contact us by email at [mailto:orders@ettus.com orders@ettus.com], or by phone at +14086106399 (Monday-Friday, 8 AM - 5 PM, Pacific Time). Please be sure to include your order number and the serial number of your USRP.<br />
<br />
==Terms and Conditions of Sale==<br />
Terms and conditions of sale can be accessed online at the following link: http://www.ettus.com/legal/terms-and-conditions-of-sale<br />
<br />
[[Category:Getting Started Guides]]<br />
[[Category:X4x0]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=Knowledge_Base&diff=5872Knowledge Base2023-09-22T15:32:02Z<p>MichaelDickens: /* UHD and USRP User Manual */ add X440</p>
<hr />
<div>Welcome to the Ettus Research Knowledge Base (KB). The KB is continuously being updated and expanded. If you have any suggestions, or do not find what you are looking for, then please [http://www.ettus.com/contact Contact Us].<br />
__NOTOC__<br />
<div class="row"><br />
<div class="col-1-3"><br />
== [[Getting Started Guides|<i class="fa fa-road"></i> Getting Started Guides]] ==<br />
<br />
'''Motherboards'''<br />
* [[B200/B210/B200mini/B205mini Getting Started Guides|B200/B210/B200mini/B205mini]]<br />
* [[Ettus USRP E300 Embedded Family Getting Started Guides|E310/E312/E313]]<br />
* [[E320 Getting Started Guide|E320]]<br />
* [[N200/N210 Getting Started Guides|N200/N210]]<br />
* [[USRP N300/N310/N320/N321 Getting Started Guide|N300/N310/N320/N321]]<br />
* [[X300/X310 Getting Started Guides|X300/X310]]<br />
* [[USRP-2974 Getting Started Guide|USRP-2974]]<br />
* [[USRP X410 Getting Started Guide|X410]]<br />
* [[USRP X440 Getting Started Guide|X440]]<br />
<br />
'''Daughterboards'''<br />
* [[BasicTX/BasicRX Getting Started Guides|BasicTX/BasicRX]]<br />
* [[CBX Getting Started Guides|CBX]]<br />
* [[LFTX/LFRX Getting Started Guides|LFTX/LFRX]]<br />
* [[SBX Getting Started Guides|SBX]]<br />
* [[TwinRX Getting Started Guides|TwinRX]]<br />
* [[UBX Getting Started Guides|UBX]]<br />
* [[WBX Getting Started Guides|WBX]]<br />
<br />
'''Other'''<br />
* [[Getting_Started_with_RFNoC_in_UHD_4.0|RFNoC Development (UHD 4.x)]]<br />
* [[RFNoC_4_Migration_Guide|RFNoC Migration Guide (UHD 3.x to UHD 4.x)]]<br />
* [[Getting_Started_with_RFNoC_Development|RFNoC Development (UHD 3.x)]]<br />
* [[Live SDR Environment Getting Started Guides|Live SDR Environment]]<br />
* [[OctoClock CDA-2990 Getting Started Guides|OctoClock CDA-2990]]<br />
* [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices|White Rabbit]]<br />
* [[Getting Started with DPDK and UHD|DPDK]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Hardware Resources|<i class="fa fa-cogs"></i> Hardware Resources]] ==<br />
'''Motherboards'''<br />
* [[B200/B210/B200mini/B205mini]]<br />
* [[Ettus USRP E300 Embedded Family Hardware Resources|E310/E312/E313]]<br />
* [[E320|E320]]<br />
* [[N200/N210]]<br />
* [[N300/N310]]<br />
* [[N320/N321]]<br />
* [[X300/X310]]<br />
* [[USRP-2974]]<br />
* [[X410]]<br />
* [[X440]]<br />
<br />
'''Daughterboards'''<br />
* [[BasicTX/BasicRX]]<br />
* [[CBX]]<br />
* [[LFTX/LFRX]]<br />
* [[SBX]]<br />
* [[TwinRX]]<br />
* [[UBX]]<br />
* [[WBX]]<br />
<br />
'''Other'''<br />
* [[OctoClock CDA-2990]]<br />
* [[GPSDO]]<br />
* [[Antennas]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Software Resources|<i class="fa fa-desktop"></i> Software Resources]] ==<br />
'''Ettus Products'''<br />
* [[UHD]]<br />
* [[UHD Python API]]<br />
* [[Getting_Started_with_RFNoC_in_UHD_4.0|RFNoC (UHD 4.x)]]<br />
* [[RFNoC]] (UHD 3.x)<br />
<br />
'''Third Party'''<br />
* [[GNU Radio]]<br />
* [[LabVIEW]]<br />
* [[Matlab/Simulink]]<br />
* [[OpenBTS]]<br />
* [[Eurecom OpenAirInterface (OAI)]]<br />
* [[srsLTE/srsUE]]<br />
* [[Gqrx]]<br />
* [[Fosphor]]<br />
<br />
'''Reference Architectures'''<br />
* [[Multichannel RF Reference Architecture]]<br />
<br />
</div><br />
</div><br />
<br />
<div class="row"><br />
<div class="col-1-3"><br />
<br />
== [[UHD and USRP User Manual|<i class="fa fa-flag"></i> UHD and USRP User Manual]] ==<br />
<br />
'''Software'''<br />
* [https://files.ettus.com/manual/ UHD Manual (master)]<br />
* [https://files.ettus.com/manual_archive/ UHD Manual Archive (previous releases)]<br />
<br />
'''Motherboards'''<br />
* [https://files.ettus.com/manual/page_usrp_b200.html B200/B210/B200mini/B205mini]<br />
* [https://files.ettus.com/manual/page_usrp_x3x0.html X300/X310]<br />
* [https://files.ettus.com/manual/page_usrp2.html N200/N210]<br />
* [https://files.ettus.com/manual/page_usrp_n3xx.html N300/N310/N320/N321]<br />
* [https://files.ettus.com/manual/page_usrp_e3xx.html E310/E312/E313/E320]<br />
* [https://files.ettus.com/manual/page_usrp_x4xx.html X410/X440]<br />
<br />
<br />
'''Daughterboards'''<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_basictx BasicRX/LFRX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_basicrx BasicTX/LFTX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_cbx CBX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_sbx SBX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_wbx WBX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_ubx UBX]<br />
* [https://files.ettus.com/manual/page_dboards.html#dboards_twinrx TwinRX]<br />
<br />
'''Other'''<br />
* [https://files.ettus.com/manual/page_octoclock.html OctoClock]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Application Notes|<i class="fa fa-file-text-o"></i> Application Notes]] ==<br />
Application Notes (AN) and technical articles written by engineers, for engineers. These articles offer experienced analysis, design ideas, reference designs, and tutorials—to make you productive and successful using USRP devices.<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Additional Resources|<i class="fa fa-book"></i> Additional Resources]] ==<br />
* [[Training]]<br />
* [[Suggested Reading|Suggested Reading]]<br />
* [[Suggested Videos|Suggested Videos]]<br />
* [[SDR Events]]<br />
* [[CGRAN]]<br />
</div><br />
</div><br />
<br />
<br />
<div class="row"><br />
<div class="col-1-3"><br />
<br />
== [[Technical Support|<i class="fa fa-life-ring"></i> Technical Support]] ==<br />
* [[Email|Email]]<br />
* [[Mailing Lists|Mailing Lists]]<br />
* [[Matrix|Matrix]]<br />
* [[Internet Relay Chat (IRC)|Internet Relay Chat (IRC)]]<br />
* [[StackExchange|StackExchange]]<br />
* [[Ordering and Fulfillment Help | Ordering and Fulfillment Help]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Faq|<i class="fa fa-info-circle"></i> FAQ]] ==<br />
* [[Technical FAQ|Technical]]<br />
* [[Licensing FAQ|Licensing]]<br />
<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Legacy Products| <i class="fa fa-hourglass-end"></i> Legacy Products]] ==<br />
'''Motherboards'''<br />
* [[USRP1|USRP1]]<br />
* [[USRP2|USRP2]]<br />
* [[E100/E110|E100/E110]]<br />
* [[B100]]<br />
<br />
'''Daughterboards'''<br />
* [[DBSRX2]]<br />
* [[TVRX2]]<br />
* [[XCVR2450]]</div>MichaelDickenshttps://kb.ettus.com/index.php?title=About_Sampling_Rates_and_Master_Clock_Rates_for_the_USRP_X440&diff=5871About Sampling Rates and Master Clock Rates for the USRP X4402023-09-22T15:21:30Z<p>MichaelDickens: add AN number & authors inf</p>
<hr />
<div><!-- Title: About Sampling Rates and Master Clock Rates for the USRP X440 --><br />
<br />
== Application Note Number and Authors ==<br />
<br />
'''AN-055''' by Marian Koop<br />
<!-- Internal use only: please do keep this updated!<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2023-09-22<br />
|style="text-align:center;"| Marian Koop <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<span id="overview"></span><br />
= Overview = <br />
This application note guides users through the selection process of Master Clock Rates (MCR) for the [https://kb.ettus.com/X440#X440 USRP X440]. It will highlight possible implications and side effects as well as design specific differences to other USRPs (like the X410).<br />
<br />
<span id="x440-cfg_considerations"></span><br />
= USRP X440 Configuration Considerations = <br />
The USRP X440 is a [https://uhd.readthedocs.io/en/latest/page_fbx.html#fbx_too balun-coupled transceiver] without built-in RF signal conditioning. Compared to other RF architectures this enables the USRP X440 to access the full RF bandwidth available to the ADC/DAC, but also requires additional frequency planning. To achieve this, the USRP X440 utilizes its ADC/DAC in direct sampling mode and is susceptible to various effects that may distort the signal of interest. These can be separated into distortions from both signal processing and from the ADC/DAC design.<br />
<br />
<span id="converter_rate-mcr-iq_rate"></span><br />
== Relationship between RF-ADC/DAC Converter Rate, USRP Master Clock Rate (MCR), Data IQ rate ==<br />
The default USRP X440 FPGA images do not contain a configurable DDC<ref>Digital Down Conversion</ref>/DUC<ref>Digital Up Conversion</ref> block. This means that the IQ sample rate (F<sub>IQ</sub>) is the same as the Master Clock Rate (MCR) that goes into the RFNoC Radio block. However, unlike most other USRPs the USRP X440 supports a highly variable MCR. The RF Data Converter sampling rate (F<sub>S</sub>) is chosen by UHD based on the MCR and the available resampling factors of 2, 4, or 8, and defaults to the highest achievable values with these factors. This can be overridden if the desired MCR can be achieved with multiple converter rates (see device argument [https://uhd.readthedocs.io/en/latest/page_usrp_x4xx.html#x4xx_usage_args converter_rate]). Figure 1 depicts the simplified signal path block diagram for the USRP X440.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. Simplified USRP X440 Signal Path<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-X440_signal_path.jpg|450px|center]]<br />
|}<br />
<br />
<span id="dsp-distortions"></span><br />
== Aliases and Nyquist Zones ==<br />
By itself the USRP X440 can sample input signals at frequencies above the Nyquist frequency<ref>[https://en.wikipedia.org/wiki/Nyquist_frequency Nyquist frequency]</ref>, which is half of the ADC converter sampling rate (F<sub>S</sub>). However, this method introduces aliasing effects, which cause unwanted signals to appear as mirror images around multiples of the Nyquist frequency (F<sub>S</sub>/2) in the output spectrum. The first Nyquist zone (N1) is the frequency range from 0 to F<sub>S</sub>/2, and the second Nyquist zone (N2) goes from F<sub>S</sub>/2 to F<sub>S</sub>. Other Nyquist zones are numbered in ascending order, each spanning F<sub>S</sub>/2. The digital passband in each Nyquist zone can be calculated as 0.4 * F<sub>S</sub> (see also <ref>[https://docs.xilinx.com/r/en-US/pg269-rf-data-converter/RF-ADC-Decimation-Filters-Gen-1/Gen-2 Xilinx RF-ADC Decimation Filters (Gen-1)]</ref>). The following figure depicts the Nyquist zones for the minimum and maximum RF-ADC converter rates supported by the USRP X440. Note that the illustrations do not show the effects of external, analog filters on the achievable passband within a Nyquist zone. A typical expectation is, that the unusable frequency range around each Nyquist zone boundary (also often referred to as a guard band) increases with ascending Nyquist zone order and results in decreasing, lopsided achievable passbands.<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Nyquist Zones<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-X440_nyquist_zones.jpg|450px|center]]<br />
|}<br />
<br />
Knowledge of signal aliases can both be exploited and create uncertainties. Applications could utilize intentional under sampling (also referred as "bandpass sampling") to receive signals at greater than F<sub>S</sub>/2. The same effect may on the other hand lead to garbled or distorted signal detection if the signal of interest spans multiple Nyquist zones or interferer signals are aliased into the observed spectrum. Both effects are depicted in figure 3.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Aliases - Wanted and unwanted<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-wanted_and_unwanted_aliases.jpg|450px|center]]<br />
|}<br />
<br />
Applications should therefore prefer converter rates that can contain the desired signal spectrum in a single Nyquist zone, or split the signal spectrum among multiple channels and devices. While the USRP X440 does not limit the utilized Nyquist zone, performance degrades in higher orders zones and application should focus on operating in Nyquist zones 1 and 2 (For RX, zone 3 is possible, but zones 4 and higher result into significant performance degradation).<br />
<br />
<span id="adc-distortions"></span><br />
== RF-ADC Spurs to consider and how to predict them ==<br />
Another kind of distortion originates from the ADC/DAC itself. The USRP X440 uses the Xilinx RFDC, which is a design that combines multiple converters to <br />
achieve high RF-ADC rates. An RF-ADC in this design has 8 sub-ADCs that are interleaved together. The resulting offset spurs are minimized by the integrated self-calibration (see also <ref>[https://docs.xilinx.com/r/en-US/pg269-rf-data-converter/Key-CAL-Features-and-Guidance-Summary Key CAL Features and Guidance Summary]</ref>) executed by UHD but may still be detectable in the signal spectrum.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. RF-ADC Spurs with MCR = 500 MHz<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-adc_distortions.jpg|450px|center]]<br />
|}<br />
<br />
Even with the best calibration the RF-ADC spurs may still be detectable in the captured spectrum. Knowledge of the location of the spurs, in particular the RF-ADC offset spur may be used during frequency planning to select an MCR (and converter rate) that exclude the offset spur frequencies from the capture spectrum. RF-ADC input spurs on the other hand will be more difficult to avoid, but as rule of thumb for modulated input signals carrier frequencies that fall on an RF-ADC offset spur frequency should be avoided (because offset and input spurs would superimpose each other).<br />
<br />
=== How to predict offset spurs ===<br />
Any residual DC offset not corrected appears as a spur at k*F<sub>S</sub>/N, where F<sub>S</sub> is the composite converter rate of the RF-ADC, N is the number of sub-RF-ADCs interleaved together (8 for X4xx devices), and k = 0, 1, 2, … N.<br />
For more information on expected spurs levels, refer to OIS at <ref>[https://docs.xilinx.com/r/en-US/ds926-zynq-ultrascale-plus-rfsoc/RF-ADC-Performance-Characteristics RF-ADC Performance Characteristics]</ref>.<br />
<br />
=== How to predict input spurs ===<br />
Any residual difference from gain and time skew correction results in spurious signals at +/-f<sub>in</sub> + (k/N)*F<sub>S</sub>, where F<sub>S</sub> is the converter rate of the RF-ADC, N is the number of sub-RF-ADCs (8 for X4xx devices), and f<sub>in</sub> is the frequency of the input signal.<br />
For more information on expected spurs levels, refer to GTIS at <ref>[https://docs.xilinx.com/r/en-US/ds926-zynq-ultrascale-plus-rfsoc/RF-ADC-Performance-Characteristics RF-ADC Performance Characteristics]</ref>.<br />
<br />
<span id="dac-distortions"></span><br />
== RF-DAC Distortions ==<br />
Like the RF-ADC, the RF-DAC is also not an ideal circuitry and suffers from zero-order hold reconstruction. To counter this undesired attenuation in all but the first Nyquist zone, the Xilinx RF-DAC offers a Mix-Mode, which improves the power response in the second Nyquist zone and is utilized by the USRP X440. This, together with an inverse sinc filter to counter residual distortion limits the practical use of the RF-DAC to the first two Nyquist zones. For more information on the RF-DAC mix-mode and inverse sinc filter characteristics, refer to <ref>[https://docs.xilinx.com/r/en-US/pg269-rf-data-converter/RF-DAC-Nyquist-Zone-Operation RF-DAC Nyquist Zone Operation]</ref>.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5 <ref>[https://docs.xilinx.com/viewer/attachment/wIPPkVrh~0jtjy6aqWeihQ/q5kXcl5Z7lFon2v535oW0w Xilinx: Ideal DAC Output Response, Normalised to Fsample]</ref>. RF-DAC Mix-Mode and normal, ideal roll-off sinc response.<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-dac_roll-off_sinc_response.png|350px|center]]<br />
|}<br />
<br />
<!--<br />
source: https://docs.xilinx.com/viewer/attachment/wIPPkVrh~0jtjy6aqWeihQ/q5kXcl5Z7lFon2v535oW0w<br />
--><br />
<br />
<span id="use-cases"></span><br />
= Use cases =<br />
== Scan Spectrum within single Nyquist zone ==<br />
=== Spectrum Capture between 1.7 and 1.9 GHz. ===<br />
* Min Bandwidth: 200 MHz<br />
* Minimum IQ Rate: 250 MSps<br />
==== Option 1: MCR = 250 MHz, RF-ADC converter rate = 2 GHz ====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-1700-1800_250e6_2000e6.jpg|450px|center]]<br />
|}<br />
This is not the best option because our spectrum of interest is very close to a Nyquist zone boundary.<br />
<br />
==== Option 2: MCR = 300 MHz, RF-ADC converter rate = 2.4 GHz ====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-1700-1800_300e6_2400e6.jpg|450px|center]]<br />
|}<br />
This option is better, the spectrum of interest is well within a Nyquist zone. Within the bandwidth of interest falls one of the ADC offset spurs. If the input signal is relatively strong, the impact from this small spur is negligible. If on the other hand the input signal strength is on the low end (for the X440), then maybe a different MCR should be considered.<br />
<br />
==== Option 3: MCR = 320 MHz, RF-ADC converter rate = 2.56 GHz ====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-1700-1800_320e6_2560e6.jpg|450px|center]]<br />
|}<br />
This option avoids a potential impact of an ADC offset spur in the observed spectrum. The only drawback to option 2 is the slightly higher data rate during host post-processing.<br />
<br />
=== Spectrum Capture between 500 MHz and 1.5 GHz. ===<br />
The USRP X440 ships with multiple [https://uhd.readthedocs.io/en/latest/page_usrp_x4xx.html#x4xx_updating_fpga_types FPGA image flavors]. These either support 400 MHz or 1600 MHz RF bandwidth per channel. To address this use case, users have the option of using a bit file with 1600 MHz RF bandwidth to capture a contiguous spectrum, or use a 400 MHz bit file and create a stitched spectrum during host side post-processing. <br />
==== Using 1600 MHz image ====<br />
* Min Bandwidth: 1000 MHz<br />
* Minimum IQ Rate: 1250 MSps<br />
===== Option 1: MCR = 1280 MHz, RF-ADC converter rate = 2.56 GHz =====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-500-1500_1250e6_2560e6.jpg|450px|center]]<br />
|}<br />
Bad option, because desired spectrum spans multiple Nyquist zones.<br />
<br />
===== Option 2: MCR = 1600 MHz, RF-ADC converter rate = 3.2 GHz =====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-500-1500_1600e6_3200e6.jpg|450px|center]]<br />
|}<br />
This is not the best option because our spectrum of interest is very close to a Nyquist zone boundary.<br />
<br />
===== Option 3: MCR = 1689.6 MHz, RF-ADC converter rate = 3.3792 GHz =====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-500-1500_1689e6_3379e6.jpg|450px|center]]<br />
|}<br />
This option would work. If the input signal strength is low, using an even higher MCR would reduce the number of potential ADC offset spurs. This needs to be traded off against a higher data rate during host post-processing.<br />
<br />
==== Using 400 MHz image ====<br />
Due to the smaller bandwidth addressing the use case will require the use of multiple channels. The captured spectra than needs to be stitched (combined) together.<br />
* Max Bandwidth: 400 MHz<br />
===== Option 1: MCR 400 MHz, RF-ADC converter rate = 3.2 GHz, 3 channels =====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-500-1500_400e6_3200e6.jpg|450px|center]]<br />
|}<br />
This is not the best option because our spectrum of interest boundary is very close to a Nyquist zone boundary.<br />
<br />
===== Option 2: MCR 450 MHz, RF-ADC converter rate = 3.6 GHz, 3 channels =====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-500-1500_450e6_3600e6.jpg|450px|center]]<br />
|}<br />
This option would work. Since the spectrum of interest is larger than the per channel bandwidth the captured spectrum may contain potential ADC offset spurs. For the spectrum of interest the 3 channels would nicely use tune frequencies (680, 1000, 1320 MHz) that do not match ADC offset spur frequencies.<br />
<br />
===== Option 3: MCR 512 MHz, RF-ADC converter rate = 4.096 GHz, 3 channels =====<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|-<br />
| style="border-style: none;" | [[File:ANxxx-500-1500_512e6_4096e6.jpg|450px|center]]<br />
|}<br />
This option would work as well. Like in option 2, the spectrum of interest may contain one potential ADC offset spur (compared to 2 in option 2). The drawback to option 2 is of course again the higher data rate during host post-processing that may be offset by only having to stitch less spectra (2 vs 3) together.<br />
<br />
<!--<br />
== Scan Spectrum that spans multiple Nyquist Zones ==<br />
<br />
=== Tradeoffs: spectrum hole vs. use multiple mcrs/devices ===<br />
--><br />
<br />
<br />
<span id="appendix"></span><br />
= Appendix =<br />
<span id="x440-mcrs"></span><br />
== X440 Supported Master Clock Rates (MCR) ==<br />
Note: The selected FPGA bitfile may further limit the maximum supported master clock rate.<br />
{| class="wikitable" style="text-align: center; margin:auto"<br />
|-<br />
! MCR (MHz) !! RFDC Converter Rate (GHz)<br />
|-<br />
| 125.0 || 1.0<br />
|-<br />
| 128.0 || 1.024<br />
|-<br />
| 133.12 || 1.06496<br />
|-<br />
| 150.0 || 1.2<br />
|-<br />
| 153.6 || 1.2288<br />
|-<br />
| 160.0 || 1.28<br />
|-<br />
| 163.84 || 1.31072<br />
|-<br />
| 184.32 || 1.47456<br />
|-<br />
| 199.68 || 1.59744<br />
|-<br />
| 200.0 || 1.6<br />
|-<br />
| 204.8 || 1.6384<br />
|-<br />
| 240.0 || 1.92<br />
|-<br />
| 245.76 || 1.96608<br />
|-<br />
| 250.0 || 2.0, 1.0<br />
|-<br />
| 256.0 || 2.048, 1.024<br />
|-<br />
| 266.24 || 2.12992, 1.06496<br />
|-<br />
| 300.0 || 2.4, 1.2<br />
|-<br />
| 307.2 || 2.4576, 1.2288<br />
|-<br />
| 320.0 || 2.56, 1.28<br />
|-<br />
| 327.68 || 2.62144, 1.31072<br />
|-<br />
| 360.0 || 2.88, 1.44<br />
|-<br />
| 368.64 || 1.47456, 2.94912<br />
|-<br />
| 375.0 || 3.0, 1.5<br />
|-<br />
| 384.0 || 1.536, 3.072<br />
|-<br />
| 399.36 || 3.19488, 1.59744<br />
|-<br />
| 400.0 || 3.2, 1.6<br />
|-<br />
| 409.6 || 3.2768, 1.6384<br />
|-<br />
| 450.0 || 1.8, 3.6<br />
|-<br />
| 460.8 || 1.8432, 3.6864<br />
|-<br />
| 480.0 || 3.84, 1.92<br />
|-<br />
| 491.52 || 3.93216, 1.96608<br />
|-<br />
| 500.0 || 4.0, 2.0, 1.0<br />
|-<br />
| 512.0 || 1.024, 2.048, 4.096<br />
|-<br />
| 532.48 || 1.06496, 2.12992<br />
|-<br />
| 552.96 || 2.21184, 1.10592<br />
|-<br />
| 599.04 || 1.19808, 2.39616<br />
|-<br />
| 600.0 || 1.2, 2.4<br />
|-<br />
| 614.4 || 1.2288, 2.4576<br />
|-<br />
| 625.0 || 1.25, 2.5<br />
|-<br />
| 640.0 || 2.56, 1.28<br />
|-<br />
| 655.36 || 1.31072, 2.62144<br />
|-<br />
| 665.6 || 1.3312, 2.6624<br />
|-<br />
| 720.0 || 1.44, 2.88<br />
|-<br />
| 737.28 || 1.47456, 2.94912<br />
|-<br />
| 750.0 || 1.5, 3.0<br />
|-<br />
| 768.0 || 1.536, 3.072<br />
|-<br />
| 798.72 || 3.19488, 1.59744<br />
|-<br />
| 800.0 || 1.6, 3.2<br />
|-<br />
| 819.2 || 3.2768, 1.6384<br />
|-<br />
| 840.0 || 1.68, 3.36<br />
|-<br />
| 860.16 || 3.44064, 1.72032<br />
|-<br />
| 875.0 || 3.5, 1.75<br />
|-<br />
| 896.0 || 3.584, 1.792<br />
|-<br />
| 900.0 || 1.8, 3.6<br />
|-<br />
| 921.6 || 1.8432, 3.6864<br />
|-<br />
| 931.84 || 1.86368, 3.72736<br />
|-<br />
| 960.0 || 3.84, 1.92<br />
|-<br />
| 983.04 || 1.96608, 3.93216<br />
|-<br />
| 998.4 || 3.9936, 1.9968<br />
|-<br />
| 1000.0 || 4.0, 2.0<br />
|-<br />
| 1024.0 || 4.096, 2.048<br />
|-<br />
| 1050.0 || 2.1<br />
|-<br />
| 1064.96 || 2.12992<br />
|-<br />
| 1075.2 || 2.1504<br />
|-<br />
| 1080.0 || 2.16<br />
|-<br />
| 1105.92 || 2.21184<br />
|-<br />
| 1120.0 || 2.24<br />
|-<br />
| 1125.0 || 2.25<br />
|-<br />
| 1146.88 || 2.29376<br />
|-<br />
| 1152.0 || 2.304<br />
|-<br />
| 1198.08 || 2.39616<br />
|-<br />
| 1200.0 || 2.4<br />
|-<br />
| 1228.8 || 2.4576<br />
|-<br />
| 1280.0 || 2.56<br />
|-<br />
| 1290.24 || 2.58048<br />
|-<br />
| 1310.72 || 2.62144<br />
|-<br />
| 1331.2 || 2.6624<br />
|-<br />
| 1350.0 || 2.7<br />
|-<br />
| 1382.4 || 2.7648<br />
|-<br />
| 1397.76 || 2.79552<br />
|-<br />
| 1400.0 || 2.8<br />
|-<br />
| 1433.6 || 2.8672<br />
|-<br />
| 1440.0 || 2.88<br />
|-<br />
| 1474.56 || 2.94912<br />
|-<br />
| 1500.0 || 3.0<br />
|-<br />
| 1536.0 || 3.072<br />
|-<br />
| 1600.0 || 3.2<br />
|-<br />
| 1625.0 || 3.25<br />
|-<br />
| 1638.4 || 3.2768<br />
|-<br />
| 1650.0 || 3.3<br />
|-<br />
| 1658.88 || 3.31776<br />
|-<br />
| 1664.0 || 3.328<br />
|-<br />
| 1680.0 || 3.36<br />
|-<br />
| 1689.6 || 3.3792<br />
|-<br />
| 1720.32 || 3.44064<br />
|-<br />
| 1730.56 || 3.46112<br />
|-<br />
| 1750.0 || 3.5<br />
|-<br />
| 1760.0 || 3.52<br />
|-<br />
| 1792.0 || 3.584<br />
|-<br />
| 1797.12 || 3.59424<br />
|-<br />
| 1800.0 || 3.6<br />
|-<br />
| 1802.24 || 3.60448<br />
|-<br />
| 1843.2 || 3.6864<br />
|-<br />
| 1863.68 || 3.72736<br />
|-<br />
| 1875.0 || 3.75<br />
|-<br />
| 1920.0 || 3.84<br />
|-<br />
| 1950.0 || 3.9<br />
|-<br />
| 1966.08 || 3.93216<br />
|-<br />
| 1996.8 || 3.9936<br />
|-<br />
| 2000.0 || 4.0<br />
|-<br />
| 2027.52 || 4.05504<br />
|-<br />
| 2048.0 || 4.096<br />
|}<br />
<br />
<br />
<span id="references"></span><br />
== References and Related Documentation ==<br />
* [https://www.ni.com/en/solutions/aerospace-defense/radar-electronic-warfare-sigint/advantages-of-direct-rf-sampling-architectures.html Advantages of Direct RF Sampling Architectures]<br />
* [https://docs.xilinx.com/r/en-US/pg269-rf-data-converter Zynq UltraScale+ RFSoC RF Data Converter v2.6 Gen 1/2/3/DFE LogiCORE IP Product Guide (PG269)]<br />
* [https://events.gnuradio.org/event/21/contributions/392/attachments/123/285/Lo%20and%20behold,%20no%20LO.pdf GRcon 23 - Lo and behold, no LO!]<br />
* [https://docs.xilinx.com/r/en-US/ds926-zynq-ultrascale-plus-rfsoc/RF-ADC-Electrical-Characteristics Zynq UltraScale+ RFSoC Data Sheet: DC and AC Switching Characteristics (DS926)]<br />
<br />
<br />
= Footnotes =<br />
<references /></div>MichaelDickenshttps://kb.ettus.com/index.php?title=Application_Notes&diff=5870Application Notes2023-09-22T14:40:55Z<p>MichaelDickens: Add AN-055</p>
<hr />
<div>Application Notes (AN) and technical articles written by engineers, for engineers. These articles offer experienced analysis, design ideas, reference designs, and tutorials—to make you productive and successful using USRP devices.<br />
<br />
{| class="wikitable"<br />
!colspan="4"|Application Notes<br />
|-<br />
<br />
<br />
! style="text-align:center;"| Number<br />
! style="text-align:center;"| Title<br />
! style="text-align:center;"| Abstract<br />
! style="text-align:center;"| Author(s)<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-009<br />
|style="width: 30%;"| [[Declaration of Conformity]]<br />
|style="width: 50%;"| This application note describes how to find Declaration of Conformity information for a given Ettus device. <br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-010<br />
|style="width: 30%;"| [[Trade Compliance and Export Control Classification Number (ECCN)]]<br />
|style="width: 50%;"| This application note describes how to find trade compliance information including the ECCN for a given Ettus device. <br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-055<br />
|style="width: 30%;"| [[About Sampling Rates and Master Clock Rates for the USRP X440]]<br />
|style="width: 50%;"| This application note guides users through the selection process of Master Clock Rates (MCR) for the USRP X440.<br />
|style="width: 10%; text-align: center;"| Marian Koop<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-088<br />
|style="width: 30%;"| [[USRP Host Performance Tuning Tips and Tricks]]<br />
|style="width: 50%;"| This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-111<br />
|style="width: 30%;"| [[UHD Device Eraser and Certificates of Volatility]]<br />
|style="width: 50%;"| This AN provides an overview of the UHD Device Eraser utility as well as links to the Certificates of Volatility for all Ettus products.<br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-117<br />
|style="width: 30%;"| [[GPSDO Selection Guide]]<br />
|style="width: 50%;"| This AN explains how to select and use a GPSDO with the USRP B-, N-, and X-series devices.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-121<br />
|style="width: 30%;"| [[Debugging FPGA images]]<br />
|style="width: 50%;"| This application note covers the basics to get you through the process of probing the signals inside an FPGA. In order to accomplish that, we will review briefly the 'Xilinx ChipScope Analyzer' and will apply it to one of our core RFNoC blocks: the RFNoC Signal generator. <br />
|style="width: 10%; text-align: center;"| Nicolas Cuervo <br> Sugandha Gupta <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-142<br />
|style="width: 30%;"| [[Transmitting DVB-S2 with GNU Radio and an USRP B210]]<br />
|style="width: 50%;"| This application note will demonstrate using an USRP B210 and the GNU Radio DTV example flowgraph to transmit a DVB-S2 video stream to an off-the-shelf satellite receiver. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-158<br />
|style="width: 30%;"| [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]]<br />
|style="width: 50%;"| This application note provides instructions for synchronizing multiple USRP N3xx devices using White Rabbit Ethernet-based synchronization. <br />
|style="width: 10%; text-align: center;"| Dan Baker<br />
Wan Liu <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-177<br />
|style="width: 30%;"| [[About USRP Bandwidths and Sampling Rates]]<br />
|style="width: 50%;"| This AN provides insight into the topics of USRP architecture, system bandwidth, host interface throughput, and available sampling rates.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-178<br />
|style="width: 30%;"| [[Resolving Audio Codec Enumeration Issues On The E31x]]<br />
|style="width: 50%;"| This application note covers Resolving Audio Codec Enumeration Issues On The E31x. <br />
|style="width: 10%; text-align: center;"| Logan Fagg<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-188<br />
|style="width: 30%;"| [[Interrogating Passive Wireless SAW Sensors with the USRP]]<br />
|style="width: 50%;"| Typical interrogator design for wireless SAW sensor systems require many discrete components and lengthy build times, making it difficult to rapidly adapt to sensor designs in a research environment. We have employed the USRP B200 as a SAW sensor interrogation system. Interrogation of wideband orthogonal frequency coded (OFC) SAW sensors imposes strict requirements on the timing and synchronization of the transceiver. The USRP FPGA has been modified to operate in a synchronous, pulsed mode of operation, allowing rapid data acquisition and the full 56MHz bandwidth to be utilized. Data from the USRP is passed to a custom matched filter correlator routine to extract sensor parameters. The system is capable of interrogating multiple sensors, simultaneously. Demonstration of the system is accomplished by wirelessly interrogating SAW sensors at 915MHz and extracting temperature.<br />
|style="width: 10%; text-align: center;"| Trip Humphries<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-204<br />
|style="width: 30%;"| [[Getting Started with UHD and C++]]<br />
|style="width: 50%;"| This AN explains how to write and build C++ programs that use the UHD API and introduces<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-244<br />
|style="width: 30%;"| [[Direction Finding with the USRP™ X-Series and TwinRX™]]<br />
|style="width: 50%;"| This application note covers using the USRP™ TwinRX™ daughterboard in a direction find application using the MUSIC algorithm. <br />
|style="width: 10%; text-align: center;"| Srikanth Pagadarai <br> Travis Collins <br> Alexander M. Wyglinski<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-296<br />
|style="width: 30%;"| [[Using Dual 10 Gigabit Ethernet on the USRP X300/X310]]<br />
|style="width: 50%;"| This short guide is meant to help in quickly setting up an X-series USRP for use over two 10 Gigabit Ethernet links simultaneously. <br />
|style="width: 10%; text-align: center;"| Paul David<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-305<br />
|style="width: 30%;"| [[X300/X310 Device Recovery]]<br />
|style="width: 50%;"| This application note covers the details of recovering the USRP X300/X310 via JTAG. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-309<br />
|style="width: 30%;"| [[About the Motherboard and Daughtercard EEPROM on USRP Devices]]<br />
|style="width: 50%;"| This AN discusses the EEPROM storage on various USRP devices and daughtercards. This guides explains how to update the EEPROM contents and recover from EEPROM corruption. The product codes, which are also stored in the EEPROM, for all USRP devices and daughtercards are also given for reference.<br />
|style="width: 10%; text-align: center;"| Trip Humphries<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-311<br />
|style="width: 30%;"| [[Software Development on the E310 and E312]]<br />
|style="width: 50%;"| This application note covers the software development process on the USRP E310 and E312. <br />
|style="width: 10%; text-align: center;"| Martin Braun <br> Nicolas Cuervo<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-315<br />
|style="width: 30%;"| [[Software Development on the E3xx USRP - Building RFNoC UHD / GNU Radio / gr-ettus from Source]]<br />
|style="width: 50%;"| This application note is one of a multi-part series which will cover the software development process on the USRP E310, E312 and E313. It will cover building the rfnoc-devel branch of UHD, GNU Radio and gr-ettus from source for the host machine, and cross-compiling the rfnoc-devel branch of UHD, GNU Radio and gr-ettus for the E3xx USRP. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-322<br />
|style="width: 30%;"| [[Experiments with the UBX Daughterboard in the HF Band]]<br />
|style="width: 50%;"| We show the results of experiments with the UBX daughtercard on an USRP X310 platform for use in the HF frequency range, from 1.8MHz to 30MHz. While the UBX is nominally rated for use only down to 10 MHz, with careful flow-graph design, and pre-filtering, it provides quite-good performance across the HF bands.<br />
|style="width: 10%; text-align: center;"| Marcus Leech<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-325<br />
|style="width: 30%;"| [[N200/N210 Device Recovery]]<br />
|style="width: 50%;"| This application note covers the details of recovering your N200/N210.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-335<br />
|style="width: 30%;"| [[Streaming processed data from the E31x with GNU Radio and ZMQ]]<br />
|style="width: 50%;"| This application note will demonstrate using the USRP E310 to remotely stream processed data to a host machine. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-355<br />
|style="width: 30%;"| [[Modifying an X310 Chassis for External LO Sharing]]<br />
|style="width: 50%;"| This document describes how to modify an X310 chassis to wire the LO out of the back plate. Doing this will allow the user to export and import an LO signal as desired when using a compatible daughterboard such as the TwinRX. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-363<br />
|style="width: 30%;"| [[Implementation of an ADS-B/Mode-S Receiver in GNU Radio]]<br />
|style="width: 50%;"| This AN guides the reader through the implementation of an ADS-B receiver using the gr-air-modes Out-of-Tree (OOT) module for GNU Radio. An explanation of ADS-B is also provided, and several real-world, over-the-air examples and profiled.<br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-400<br />
|style="width: 30%;"| [[Getting Started with RFNoC in UHD 4.0]]<br />
|style="width: 50%;"| This AN describes how use RFNoC in UHD 4.0, including building FPGA images for RFNoC, changing which blocks are included in the build, and creating your own RFNoC blocks.<br />
|style="width: 10%; text-align: center;"| Sugandha Gupta <br> Brent Stapleton <br> Wade Fife <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-401<br />
|style="width: 30%;"| [[RFNoC 4 Migration Guide]]<br />
|style="width: 50%;"| Guide on how to migrate RFNoC blocks written for RFNoC 3 to RFNoC 4.<br />
|style="width: 10%; text-align: center;"| Jonathon Pendlum<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-444<br />
|style="width: 30%;"| [[Using B200/B210/B200mini/B205mini on OSX / macOS with UHD]]<br />
|style="width: 50%;"| This AN provides a basic guide for what to expect when using a USB-based B-series USRP on OSX / macOS with UHD.<br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-445<br />
|style="width: 30%;"| [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Linux platform. Other alternate installation methods are also discussed.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-452<br />
|style="width: 30%;"| [[5G NR EVM Measurements with the USRP N320/N321]]<br />
|style="width: 50%;"| Example EVM measurements are shown using the USRP N320/N321 receiver and the 5G New Radio (5G NR) modulation standard. The use of I/Q image calibration and spur-dodging are demonstrated as methods to improve EVM performance. <br />
|style="width: 10%; text-align: center;"| Drew Fischer<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-492<br />
|style="width: 30%;"| [[Selecting a RF Daughterboard]]<br />
|style="width: 50%;"| This AN explores the RF daughterboards used by the N-series and X-series USRP devices at a high level, compares devices across several primary features, and walks the reader through the process of selecting a particular device for the their application.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-500<br />
|style="width: 30%;"| [[Getting Started with DPDK and UHD]]<br />
|style="width: 50%;"| This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
|style="width: 10%; text-align: center;"| Nate Temple <br> Alex Williams <br> Wade Fife <br> Matt Prost<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-503<br />
|style="width: 30%;"| [[Converting an X310 into an NI-USRP Rio]]<br />
|style="width: 50%;"| This Application Note explains how to use an Ettus Research-branded USRP with LabVIEW, and in effect, convert it into an NI-USRP RIO.<br />
|style="width: 10%; text-align: center;"| Tim Fountain<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-504<br />
|style="width: 30%;"| [[USRP N Series Quick Start (Daughterboard Installation)]]<br />
|style="width: 50%;"| This application note is a detailed step-by-step guide to install a daughterboard into the USRP N200/N210.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-524<br />
|style="width: 30%;"| [[Building and Installing UHD and GNU Radio in an Offline Environment]]<br />
|style="width: 50%;"| This application note will provide step-by-step instructions on building and installing UHD and GNU Radio in an offline environment. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-525<br />
|style="width: 30%;"| [[Building and Installing UHD and GNU Radio to a Custom Prefix]]<br />
|style="width: 50%;"| This application note provides step-by-step instructions on building and installing UHD and GNU Radio to a local directory. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-561<br />
|style="width: 30%;"| [[Implementation of a Simple FM Receiver in GNU Radio]]<br />
|style="width: 50%;"| This AN shows a quick and simple implementation of an FM receiver for the USRP using GNU Radio. The goal is to easily demonstrate a practical application, and to verify that the USRP is functioning properly.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-611<br />
|style="width: 30%;"| [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Windows platform.<br />
|style="width: 10%; text-align: center;"| Derek Kozel<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-620<br />
|style="width: 30%;"| [[Troubleshooting X300/X310 Device Discovery Issues]]<br />
|style="width: 50%;"| Troubleshooting guide to intended to cover some of the most commonly recommended steps to enable USRP connectivity. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-621<br />
|style="width: 30%;"| [[Troubleshooting N310/N320 Device Discovery Issues]]<br />
|style="width: 50%;"| Troubleshooting guide to intended to cover some of the most commonly recommended steps to enable USRP connectivity. Serves as a supplement to the N3xx getting started guide. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-630<br />
|style="width: 30%;"| [[Writing the USRP File System Disk Image to a SD Card]]<br />
|style="width: 50%;"| This application note will provide step-by-step instructions on writing a file system disk image to a SD card using Linux. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-638<br />
|style="width: 30%;"| [[Running UHD and GNU Radio on NI USRP-RIO]]<br />
|style="width: 50%;"| This AN explains the process to updating your NI USRP-RIO to run UHD and GNU Radio. <br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple <br> Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-642<br />
|style="width: 30%;"| [[Using the RFNoC Replay Block]]<br />
|style="width: 50%;"| This application note guides a user through basic use of the RFNoC Replay block in UHD 3.x and explains how to run the UHD Replay example. This example covers use on the X300/X310 and N310 products. <br />
|style="width: 10%; text-align: center;"| Wade Fife<br />
|-<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-642b<br />
|style="width: 30%;"| [[Using the RFNoC Replay Block in UHD 4]]<br />
|style="width: 50%;"| This application note guides a user through basic use of the RFNoC Replay block in UHD 4.x and explains how to run the UHD Replay example. <br />
|style="width: 10%; text-align: center;"| Martin Braun<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-666<br />
|style="width: 30%;"| [[Mean Time Between Failure (MTBF) of USRPs and Daughterboards]]<br />
|style="width: 50%;"| This AN provides information about the MTBF for USRPs and daughterboards<br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-725<br />
|style="width: 30%;"| [[USRP N320/N321 LO Distribution]]<br />
|style="width: 50%;"| This application note provides an overview of using the LO Distribution of the N320/N321 USRPs.<br />
|style="width: 10%; text-align: center;"| Brian Avenell <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-732<br />
|style="width: 30%;"| [[USRP E312 Battery Replacement Instructions]]<br />
|style="width: 50%;"| This application note covers replacing the battery cell inside the USRP E312. <br />
|style="width: 10%; text-align: center;"| Robin Coxe<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-788<br />
|style="width: 30%;"| [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Mac OS X platform.<br />
|style="width: 10%; text-align: center;"| Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-800<br />
|style="width: 30%;"| [[Enabling Ethernet Connectivity on Octoclock and Octoclock-G]]<br />
|style="width: 50%;"| This document supplements the UHD Manual's guide for updating the Octoclock bootloader to allow for Ethernet communications with the device. <br />
|style="width: 10%; text-align: center;"| Sam Reiter <br> Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-822<br />
|style="width: 30%;"| [[Multichannel RF Reference Architecture]]<br />
|style="width: 50%;"| This application note provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture. <br />
|style="width: 10%; text-align: center;"| Michael Dickens <br> Neel Pandeya <br> Jovian Wysocki<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-823<br />
|style="width: 30%;"| [[Getting Started with RFNoC Development]]<br />
|style="width: 50%;"| This application note gives a brief introduction into the steps required to start developing RFNoC blocks on your computer with UHD 3. <br />
|style="width: 10%; text-align: center;"| Martin Braun <br> Nicolas Cuervo<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-832<br />
|style="width: 30%;"| [[Mapping Between ER-USRP and NI-USRP Product Numbers]]<br />
|style="width: 50%;"| This application note covers the details of the mapping between Ettus Research USRP and National Instruments USRP product numbers. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-881<br />
|style="width: 30%;"| [[Selecting a USRP Device]]<br />
|style="width: 50%;"| This AN explores the USRP family at a high level, compares devices across several primary features, and walks the reader through the process of selecting a particular device for the their application.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-882<br />
|style="width: 30%;"| [[Synchronization and MIMO Capability with USRP Devices]]<br />
|style="width: 50%;"| Discusses the requirements for Multiple-In-Multiple-Out (MIMO) and phased-array systems. Summarizes the MIMO capability of each USRP device and daughterboard, and shows how to build MIMO systems with the USRP product family.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-883<br />
|style="width: 30%;"| [[Synchronizing USRP Events Using Timed Commands in UHD]]<br />
|style="width: 50%;"| Guide to cover common USRP synchronization scenarios and deep-dive into the use of timed commands within USRPs. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-888<br />
|style="width: 30%;"| [[Getting Started with 4G LTE using Eurecom OpenAirInterface (OAI) on the USRP 2974]]<br />
|style="width: 50%;"| Discusses how to install and configure the OpenAirInterface (OAI) software on the USRP 2974 hardware to implement a 4G LTE cellular basestation (eNodeB).<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-904<br />
|style="width: 30%;"| [[USRP X Series Quick Start (Daughterboard Installation)]]<br />
|style="width: 50%;"| This application note is a detailed step-by-step guide to install a daughterboard into the USRP X300/X310.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-936<br />
|style="width: 30%;"| [[Verifying the Operation of the USRP Using UHD and GNU Radio]]<br />
|style="width: 50%;"| This AN explains how to use UHD and GNU Radio, once installed, to verify the correct operation of the USRP. Several test procedures are explained in detail. Several tests make use of an optional spectrum analyzer and signal generator.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
<br />
|}</div>MichaelDickens