https://kb.ettus.com/api.php?action=feedcontributions&user=JovianWysocki&feedformat=atomEttus Knowledge Base - User contributions [en]2024-03-29T11:53:12ZUser contributionsMediaWiki 1.26.2https://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5482Multichannel RF Reference Architecture2022-08-29T14:29:35Z<p>JovianWysocki: </p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.2.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for multiple RF applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 16 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems 3U BAM'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
:o '''NVIDIA MCX4121A-ACAT ConnectX-4 Lx EN'''—Network card that supports two 25 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.2. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 systems.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software, thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimum identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
::'''Note''' For maximum streaming rates with DPDK, both SFP+0 and SFP+1 must be configured and connected. Dual port streaming can also be used without DPDK for enhanced streaming capability.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-dpdk"></span><br />
=== (Optional) Installing and Configuring DPDK ===<br />
If you want to improve streaming performance for your system, you can install Data Plane Development Kit (DPDK) 20.11. Streaming rates to disk are limited by the write speed of the chosen storage medium. Refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#streaming-simultaneously-with-dpdk-tofrom-disk-rates Streaming with DPDK Simultaneously to/from Disk Rates]'' for information on the streaming rates. <br />
<br />
::'''Note''' This reference architecture has been tested using DPDK 20.11 with NVIDIA MCX4121A-ACAT ConnectX-4 Lx EN Adapter Cards. The following steps may only apply to the NVIDIA ConnectX-4.<br />
<br />
# Ensure that you install the proper drivers for your NIC prior to installing DPDK and UHD. <br />
# Install and configure the DPDK software according to the ''[https://doc.dpdk.org/guides-20.11/linux_gsg/intro.html Data Plane Development Kit Getting Started Guide for Linux.]''<br />
# Configure huge pages by opening the grub configuration file,<br />
#: <code>/etc/default/grub</code>.<br />
# Add the following parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
#: <code>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash</code><br />
#: <code>default_hugepagesz=2048M hugepagesz=2048M hugepages=10000"</code><br />
# Close <code>/etc/default/grub</code>. <br />
# At the command prompt, update your grub configuration with the following command:<br />
#: <code>sudo update-grub</code><br />
# Reboot the host machine for these settings to take effect.<br />
# Create a UHD configuration file to interface with DPDK. <br />
#:# Run the following command to list the MAC addresses for your NICs: <br />
#:#: <code>ip a</code><br />
#:# Use the following commands to create the UHD configuration file at the location <code>/etc/uhd/</code>: <br />
#:#: <code>sudo su</code><br />
#:#: <code>mkdir -p /etc/uhd</code><br />
#:#: <code>nano /etc/uhd/uhd.conf</code><br />
# Update the following fields of your UHD configuration file.<br />
#:o Update the <code>dpdk_mac</code> field to match your NIC MAC address variables<br />
#:o Update the <code>dpdk_driver</code> field if the location is different on your system. <br />
#:o Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two-port NIC is used. There should be one core for the main DPDK thread (in this example, core #2), and separate cores assigned to each NIC (in this example, core #3 for the first port on the NIC, core #4 for the second port on the NIC). <br />
#:o Update the <code>dpdk_ipv4</code> fields to your IP range.<br />
Refer to the following <code>uhd.conf</code> file for an example of how you should update the fields in your configuration file.<br />
<nowiki><br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/dpdk-pmds<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=8192<br />
dpdk_mbuf_cache_size=64<br />
<br />
[dpdk_mac=50:6b:4b:3b:3c:d8]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.50.1/24<br />
dpdk_num_desc = 4096<br />
<br />
[dpdk_mac=50:6b:4b:3b:3c:d9]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.51.1/24<br />
dpdk_num_desc = 4096<br />
</nowiki><br />
Refer to [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Running_the_Examples ''Running the Examples''] to verify successful installation of DPDK on your system.<br />
<br />
::'''Note''' Refer to ''[https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD Getting Started with DPDK and UHD]'' for more information about DPDK and UHD. <br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' For maximum streaming rates with DPDK, dual port streaming must be configured. Follow the procedure in Updating the Network Configurations for configuring both SFP+0 and SFP+1 ports on the USRP. Ensure that the IP addresses are unique.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_txrx_fullduplex_dpdk_mem.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host memory using DPDK.<br />
:o <code>Arch_txrx_fullduplex_dpdk.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host storage using DPDK<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script: <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
:: '''Note''' If dual port streaming is configured, it must be enabled in the Reference Architecture configuration file. Uncomment <code>second_addr#</code> for each USRP and replace with the correct USRP number and SFP+1 IP address as configured in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#setting-up-the-first-usrp ''Setting Up the First USRP''].<br />
# If you are running a DPDK example, you must alter the configuration file. Add <code>use_dpdk=1</code> to the following line in the configuration file: <br />
#:<code>args = type=n3xx,master_clock_rate=250e6,use_dpdk=1</code><br />
#:: '''Note''' Ensure that you remove <code>use_dpdk=1</code> when running non-DPDK examples. <br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run <code>rfnoc_txrx_loopback.cpp</code> by executing the following command in a terminal:<br /><br />
#:<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz to 6 GHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 16 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs <ref>Measurements were taken with a two-port direct sampling analyzer. The analyzer port 0 was directly connected to USRP 0 Channel 0. Port 1 of the analyzer was connected to the output of a 16-channel RF multiplexer. All other USRP channels were connected to the input of the 16-channel RF multiplexer. USRP 0 Channel 0 was compared to each other channel sequentially.</ref> ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 16 channels<br />
| 0.009°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 16 channels<br />
| 0.019°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 16 channels<br />
| 0.034°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 16 channels<br />
| 0.070°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
| Average over 16 channels<br />
| 0.008°<br />
|-<br />
| Maximum of 16 channels<br />
| 0.018°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
| Average of 16 channels<br />
| 0.033°<br />
|-<br />
| Maximum of 16 channels<br />
| 0.070°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rates"></span><br />
=== Streaming to Memory Data Transfer Rates<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rates"></span><br />
=== Streaming to Disk Data Transfer Rates<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rates"></span><br />
=== Streaming Simultaneously to/from Disk Rates<ref>''Streaming simultaneously to/from disks'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-with-dpdk-tofrom-disk-rates"></span><br />
=== Streaming Simultaneously with DPDK to/from Disk Rates<ref>''Streaming Simultaneously with DPDK to/from Disk Rates'' Streaming simultaneously with DPDK to/from disk is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows using the Trenton Systems 3U BAM.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. Streaming Simultaneously from Host to Host Memory Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 250<br />
| 8<br />
| 7200<br />
|}<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. Streaming Simultaneously from Host to Host Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 125<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
* ''Data Plane Development Kit Getting Started Guide for Linux''—Describes how to install and configure DPDK software in a Linux environment. Visit https://doc.dpdk.org/guides-20.11/linux_gsg/intro.html.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5464Multichannel RF Reference Architecture2022-08-25T14:49:06Z<p>JovianWysocki: </p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.2.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for multiple RF applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems 3U BAM'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
:o '''NVIDIA MCX4121A-ACAT ConnectX-4 Lx EN'''—Network card that supports two 25 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.2. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 systems.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software, thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimum identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
::'''Note''' For maximum streaming rates with DPDK, both SFP+0 and SFP+1 must be configured and connected. Dual port streaming can also be used without DPDK for enhanced streaming capability.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-dpdk"></span><br />
=== (Optional) Installing and Configuring DPDK ===<br />
If you want to improve streaming performance for your system, you can install Data Plane Development Kit (DPDK) 20.11. Streaming rates to disk are limited by the write speed of the chosen storage medium. Refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#streaming-simultaneously-with-dpdk-tofrom-disk-rates Streaming with DPDK Simultaneously to/from Disk Rates]'' for information on the streaming rates. <br />
<br />
::'''Note''' This reference architecture has been tested using DPDK 20.11 with NVIDIA MCX4121A-ACAT ConnectX-4 Lx EN Adapter Cards. The following steps may only apply to the NVIDIA ConnectX-4.<br />
<br />
# Ensure that you install the proper drivers for your NIC prior to installing DPDK and UHD. <br />
# Install and configure the DPDK software according to the ''[https://doc.dpdk.org/guides-20.11/linux_gsg/intro.html Data Plane Development Kit Getting Started Guide for Linux.]''<br />
# Configure huge pages by opening the grub configuration file,<br />
#: <code>/etc/default/grub</code>.<br />
# Add the following parameters to <code>GRUB_CMDLINE_LINUX_DEFAULT</code>:<br />
#: <code>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash</code><br />
#: <code>default_hugepagesz=2048M hugepagesz=2048M hugepages=10000"</code><br />
# Close <code>/etc/default/grub</code>. <br />
# At the command prompt, update your grub configuration with the following command:<br />
#: <code>sudo update-grub</code><br />
# Reboot the host machine for these settings to take effect.<br />
# Create a UHD configuration file to interface with DPDK. <br />
#:# Run the following command to list the MAC addresses for your NICs: <br />
#:: <code>ip a</code><br />
#:# Use the following commands to create the UHD configuration file at the location /etc/uhd/: <br />
#:: <code>sudo su</code><br />
#:: <code>mkdir -p /etc/uhd</code><br />
#:: <code>nano /etc/uhd/uhd.conf</code><br />
# Update the following fields of your UHD configuration file.<br />
#:o Update the <code>dpdk_mac</code> field to match your NIC MAC address variables<br />
#:o Update the <code>dpdk_driver</code> field if the location is different on your system. <br />
#:o Update the <code>dpdk_corelist</code> and <code>dpdk_lcore</code> fields. In this example, a two-port NIC is used. There should be one core for the main DPDK thread (in this example, core #2), and separate cores assigned to each NIC (in this example, core #3 for the first port on the NIC, core #4 for the second port on the NIC). <br />
#:o Update the <code>dpdk_ipv4</code> fields to your IP range.<br />
Refer to the following <code>uhd.conf</code> file <br />
<nowiki><br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/dpdk-pmds<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=8192<br />
dpdk_mbuf_cache_size=64<br />
<br />
[dpdk_mac=50:6b:4b:3b:3c:d8]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.50.1/24<br />
dpdk_num_desc = 4096<br />
<br />
[dpdk_mac=50:6b:4b:3b:3c:d9]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.51.1/24<br />
dpdk_num_desc = 4096<br />
</nowiki><br />
Refer to [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Running_the_Examples ''Running the Examples''] to verify successful installation of DPDK on your system.<br />
<br />
::'''Note''' Refer to ''[https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD Getting Started with DPDK and UHD]'' for more information about DPDK and UHD. <br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' For maximum streaming rates with DPDK, dual port streaming must be configured. Follow the procedure in Updating the Network Configurations for configuring both SFP+0 and SFP+1 ports on the USRP. Ensure that the IP addresses are unique.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_txrx_fullduplex_dpdk_mem.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host memory using DPDK.<br />
:o <code>Arch_txrx_fullduplex_dpdk.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host storage using DPDK<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script: <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
:: '''Note''' If dual port streaming is configured, it must be enabled in the Reference Architecture configuration file. Uncomment <code>second_addr#</code> for each USRP and replace with the correct USRP number and SFP+1 IP address as configured in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#setting-up-the-first-usrp ''Setting Up the First USRP''].<br />
# If you are running a DPDK example, you must alter the configuration file. Add <code>use_dpdk=1</code> to the following line in the configuration file: <br />
#:<code>args = type=n3xx,master_clock_rate=250e6,use_dpdk=1</code><br />
#:: '''Note''' Ensure that you remove <code>use_dpdk=1</code> when running non-DPDK examples. <br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run <code>rfnoc_txrx_loopback.cpp</code> by executing the following command in a terminal:<br /><br />
#:<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 6 GHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 16 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 16 channels<br />
| 0.009°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 16 channels<br />
| 0.019°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 16 channels<br />
| 0.034°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 16 channels<br />
| 0.070°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
| Average over 16 channels<br />
| 0.008°<br />
|-<br />
| Maximum of 16 channels<br />
| 0.018°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
| Average of 16 channels<br />
| 0.033°<br />
|-<br />
| Maximum of 16 channels<br />
| 0.070°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rates"></span><br />
=== Streaming to Memory Data Transfer Rates<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rates"></span><br />
=== Streaming to Disk Data Transfer Rates<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rates"></span><br />
=== Streaming Simultaneously to/from Disk Rates<ref>''Streaming simultaneously to/from disks'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-with-dpdk-tofrom-disk-rates"></span><br />
=== Streaming Simultaneously with DPDK to/from Disk Rates<ref>''Streaming Simultaneously with DPDK to/from Disk Rates'' Streaming simultaneously with DPDK to/from disk is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows using the Trenton Systems 3U BAM.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. Streaming Simultaneously from Host to Host Memory Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 250<br />
| 8<br />
| 7200<br />
|}<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. Streaming Simultaneously from Host to Host Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 125<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
* ''Data Plane Development Kit Getting Started Guide for Linux''—Describes how to install and configure DPDK software in a Linux environment. Visit https://doc.dpdk.org/guides-20.11/linux_gsg/intro.html.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5462Multichannel RF Reference Architecture2022-08-22T20:31:32Z<p>JovianWysocki: Updated documentation for 1.2 release</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.2.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for multiple RF applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems 3U BAM'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
:o '''NVIDIA MCX4121A-ACAT ConnectX-4 Lx EN'''—Network card that supports two 25 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 systems.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimum identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
::'''Note''' For maximum streaming rates with DPDK, both SFP+0 and SFP+1 must be configured and connected. Dual port streaming can also be used without DPDK for enhanced streaming capability.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-dpdk"></span><br />
=== (Optional) Installing and Configuring DPDK ===<br />
If you want to improve streaming performance for your system, you can install Data Plane Development Kit (DPDK) 20.11. Streaming rates to disk are limited by the write speed of the chosen storage medium. Refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#streaming-simultaneously-with-dpdk-tofrom-disk-rates Streaming with DPDK Simultaneously to/from Disk Rates]'' for information on the streaming rates. <br />
<br />
::'''Note'''This reference architecture has been tested using DPDK 20.11 with NVIDIA MCX4121A-ACAT ConnectX-4 Lx EN Adapter Cards. The following steps may only apply to the NVIDIA ConnectX-4.<br />
<br />
# Ensure that you install the proper drivers for your NIC prior to installing DPDK and UHD. <br />
# Install and configure the DPDK software according to the ''[https://doc.dpdk.org/guides-20.11/linux_gsg/intro.html Data Plane Development Kit Getting Started Guide for Linux.]''<br />
# Configure huge pages by opening the grub configuration file,<br />
#: <code>/etc/default/grub</code>.<br />
# Add the following parameters to GRUB_CMDLINE_LINUX_DEFAULT:<br />
#: <code>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash</code><br />
#: <code>default_hugepagesz=2048M hugepagesz=2048M hugepages=10000</code><br />
# Close <code>/etc/default/grub</code>. <br />
# At the command prompt, update your grub configuration with the following command:<br />
#: <code>sudo update-grub</code><br />
# Reboot the host machine for these settings to take effect.<br />
# Create a UHD configuration file to interface with DPDK. <br />
#:o Run the following command to list the MAC addresses for your NICs: <br />
#:: <code>ip a</code><br />
#:o Use the following commands to create the UHD configuration file at the location /etc/uhd/: <br />
#:: <code>sudo su</code><br />
#:: <code>mkdir -p /etc/uhd</code><br />
#:: <code>nano /etc/uhd/uhd.conf</code><br />
# Update the following fields of your UHD configuration file.<br />
#:o Update the ''dpdk_mac'' field to match your NIC MAC address variables<br />
#:o Update the ''dpdk_driver'' field if the location is different on your system. <br />
#:o Update the ''dpdk_corelist'' and ''dpdk_lcore'' fields. In this example, a two-port NIC is used. There should be one core for the main DPDK thread (in this example, core #2), and separate cores assigned to each NIC (in this example, core #3 for the first port on the NIC, core #4 for the second port on the NIC). <br />
#:o Update the dpdk_ipv4 fields to your IP range.<br />
Refer to the following ''uhd.conf'' file <br />
<nowiki><br />
[use_dpdk=1]<br />
dpdk_mtu=9000<br />
dpdk_driver=/usr/local/lib/dpdk-pmds<br />
dpdk_corelist=2,3,4<br />
dpdk_num_mbufs=8192<br />
dpdk_mbuf_cache_size=64<br />
<br />
[dpdk_mac=50:6b:4b:3b:3c:d8]<br />
dpdk_lcore = 3<br />
dpdk_ipv4 = 192.168.50.1/24<br />
dpdk_num_desc = 4096<br />
<br />
[dpdk_mac=50:6b:4b:3b:3c:d9]<br />
dpdk_lcore = 4<br />
dpdk_ipv4 = 192.168.51.1/24<br />
dpdk_num_desc = 4096<br />
</nowiki><br />
Refer to [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Running_the_Examples ''Running the Examples''] to verify successful installation of DPDK on your system.<br />
<br />
::'''Note''' Refer to Getting Started with DPDK and UHD for more information about DPDK and UHD. <br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' For maximum streaming rates with DPDK, dual port streaming must be configured. Follow the procedure in Updating the Network Configurations for configuring both SFP+0 and SFP+1 ports on the USRP. Ensure that the IP addresses are unique.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_txrx_fullduplex_dpdk_mem.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host memory using DPDK.<br />
:o <code>Arch_txrx_fullduplex_dpdk.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host storage using DPDK<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
:: '''Note''' If dual port streaming is configured, it must be enabled in the Reference Architecture configuration file. Uncomment <code>second_addr#</code> for each USRP and replace with the correct USRP number and SFP+1 IP address as configured in ''Setting Up the First USRP''.<br />
# If you are running a DPDK example, you must alter the configuration file. Add <code>use_dpdk=1</code> to the following line in the configuration file: <br />
#:<code>args = type=n3xx,master_clock_rate=250e6,use_dpdk=1</code><br />
#:: '''Note''' Ensure that you remove use_dpdk=1 when running non-DPDK examples. <br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
#:<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 16 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 16 channels<br />
| 0.009°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 16 channels<br />
| 0.019°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 16 channels<br />
| 0.034°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 16 channels<br />
| 0.070°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
| Average over 16 channels<br />
| 0.008°<br />
|-<br />
| Maximum of 16 channels<br />
| 0.018°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
| Average of 16 channels<br />
| 0.033°<br />
|-<br />
| Maximum of 16 channels<br />
| 0.070°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rates"></span><br />
=== Streaming to Memory Data Transfer Rates<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rates"></span><br />
=== Streaming to Disk Data Transfer Rates<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rates"></span><br />
=== Streaming Simultaneously to/from Disk Rates<ref>''Streaming simultaneously to/from disks'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-with-dpdk-tofrom-disk-rates"></span><br />
=== Streaming Simultaneously with DPDK to/from Disk Rates<ref>''Streaming Simultaneously with DPDK to/from Disk Rates'' Streaming simultaneously with DPDK to/from disk is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows using the Trenton Systems 3U BAM.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. Streaming Simultaneously from Host to Host Memory Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 250<br />
| 8<br />
| 7200<br />
|}<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. Streaming Simultaneously from Host to Host Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 125<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
* ''Data Plane Development Kit Getting Started Guide for Linux''—Describes how to install and configure DPDK software in a Linux environment. Visit https://doc.dpdk.org/guides-20.11/linux_gsg/intro.html.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5411Multichannel RF Reference Architecture2022-04-26T20:02:48Z<p>JovianWysocki: /* Architecture Overview */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for multiple RF applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Open_Architecture_For_Radar_and_EW_Research&diff=5376Open Architecture For Radar and EW Research2022-04-20T13:47:22Z<p>JovianWysocki: Redirected page to Multichannel RF Reference Architecture</p>
<hr />
<div>#REDIRECT [[Multichannel RF Reference Architecture]]<br />
== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
== Overview ==<br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Open Architecture for Radar and EW Research. [[Media:Open_Architecture_For_Radar_and_EW_Research_v1.0.pdf | A PDF version of this AppNote is available]] for those who need or prefer that document style.<br />
<br />
This user manual describes how to design a system that enables synchronization between Receive (Rx) channels and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. The purpose of this document is to provide a starting point in the creation of your specific system configuration.<br />
<br />
== Architecture Overview ==<br />
<br />
The Open Architecture for Radar and EW Research allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO) , PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* User manual, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx and Tx-Rx synchronization.<br />
<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
<br />
* Documentation that lists installation and configuration information, best practices, and known issues encountered during development and validation of an 32x32 system. ''nxn'' refers to systems where ''n'' is the number of transmit and receive channels.<br />
<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels and 32 Tx-Rx channels.<br />
:o Streaming IQ data from 32 channels to a server for further processing.<br />
<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates waveform generation from file using RFNoC replay blocks.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Open Architecture for Radar and EW Research, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/''] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals — PPS, 10 MHz, and Local Oscillator (LO) — shown in Figure 3:<br />
<br />
* '''USRP N321''' — Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320''' — Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO''' — Clock distribution device called an OctoClock that generates and distributes 10 MHz reference clock and PPS signals and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01) — Torque screwdriver recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Open Architecture for Radar and EW Research systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR''' — Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U''' — Server with Intel dual 3rd generation Xeon Ice Lake processors and 11 Gen 4.0 PCIe slots (5–x16, 6–x8). 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems. <br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4''' — Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2''' — Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505''' — PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS''' — NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+''' — DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+''' — High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01) — Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01) — SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5) — Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket''' — Bracket required to mount the USRPs in a rack to meet environmental specifications. Contact NI for more information about how to procure Z-brackets.<br />
<br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a reference library and examples, written in C++, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written so it is scalable for systems beyond the validated 32x32 system.<br />
<br />
The software demonstrates a Tx-Rx loopback across a system of USRPs. To maximize Rx bandwidth, the replay block is used to reduce host processing and network traffic for Tx. Data moves through the system as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
= Building the System =<br />
<br />
The following figure shows a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* While running the system in loopback, attach 30dB attenuators to all output channels to prevent damage to the USRPs.<br />
<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs. Contact NI for more information about how to procure the Z-brackets.<br />
<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
<br />
* Number and label the USRPs with serial numbers. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Open_Architecture_For_Radar_and_EW_Research#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Open_Architecture_For_Radar_and_EW_Research#Connecting_the_LO_Distribution ''Connecting the LO Distribution''].<br />
<br />
* Keep signal path of all Rx, Tx, LO, PPS, and 10 MHz signals identical, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Open_Architecture_For_Radar_and_EW_Research#Cabling_the_Hardware ''Cabling the Hardware''].<br />
<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
=== Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
{{Warning|1=30dB attenuators must be used on every TX channel to ensure safe use of the MIMO Loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs}}<br />
Use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution. While connecting the LO distribution, adhere to the following:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The LO Distribution Knowledge Base article at https://kb.ettus.com/USRP_N320/N321_LO_Distribution does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Sources and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to configure and update the file system on the USRPs. The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/''] folder for a customizable template.<br />
<br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/''] folder for a customizable template.<br />
<br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
== Setting Up and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config — Contains script files that assist with the software installation process.<br />
* docs — Contains system bill of materials and reference architecture template.<br />
* examples — Contains reference architecture example and configuration files.<br />
* lib — Contains the library source files.<br />
* tools — Contains Python-based data file generator and viewer.<br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''] at [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide]. Follow the procedure with the specific steps listed in this document.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
#: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. <br />
#: For a script that will update all USRPs connected to the system, see refarch-multich-dev/config/usrp_filesystem.sh on the github repository. <br />
# Follow the procedure in ''Updating the Network Configurations''. Set the MTU to '''9000'''. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.<br />
#: '''Note''' NI strongly recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/''] folder for a customizable template. NI recommends that you assign static IP addresses for the management port IPs. This procedure is not included in this documentation.<br />
#: For a script to update the sfp1 port on all USRPs automatically, see refarch-multich-dev/config/usrp_ip_config.sh on the github repository. The host will still need to be configured manually as shown in step 5. <br />
# Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br />
#: <code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br />
#: where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated; you can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.<br />
#: '''Note''' For a script to update the FPGA images on all USRPs connected to the system, see refarch-multich-dev/config/usrp_image_loader.sh on the github repository. <br />
# Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.<br />
#: '''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards (other operating systems may configure the network interfaces differently). Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. The ports on the host require an IP address that matches the USRP it connects to. Once the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory (overwriting <code>01-network-manager-all.yaml</code>, if necessary). Then use the following command to apply the network configuration:<br />
#: <code>$ sudo netplan apply</code><br />
# Follow the procedure in ''Verifying Device Operation''.<br />
# (Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.<br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the complete procedure in [https://kb.ettus.com/Open_Architecture_For_Radar_and_EW_Research#Setting_Up_the_First_USRP ''Setting Up the First USRP''] for the remaining USRPs with the specific details listed here.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations'' — Each USRP port must be configured to have a different subnet.<br />
:o ''Updating the FPGA Image''<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
:: '''Note''' Increasing an interface's ring buffer results in more memory reserved. When setting the ring buffer make sure to monitor your memory usage. High number of interfaces can result in hundreds of GB of memory usage. A restart might be needed to clear the memory allocation.<br />
<br />
* Increase the ring buffer for each interface by calling the following command:<br />
:: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP; you can find this interface through the Network Manager or by calling the command <code>ifconfig</code>.<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs with 16 interfaces runs 16 times and results in ~65GB of memory usage. A script has been provided that can assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in [https://kb.ettus.com/Open_Architecture_For_Radar_and_EW_Research#Setting_Up_the_First_USRP ''Setting Up the First USRP''].<br />
* Enable hyperthreading on the server. Refer to server BIOS settings for information.<br />
* Enable C states on the server. Refer to server BIOS settings for information.<br />
<br />
= Using the Software =<br />
<br />
== Getting Started Examples ==<br />
<br />
Open Architecture for Radar and EW Research is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Examples''' — NI recommends that users who are new to USRP Hardware Driver or the RFNoC API begin with the following UHD RFNoC examples, located in the UHD installation folder, <code>uhd/host/examples</code>:<br />
:o <code>rfnoc_radio_loopback.cpp</code> — This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code> — This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code> — This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Examples''' — The examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/examples/''] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>iterative_loopback_singlethread.cpp</code> — This example cycles through each Tx channel to all Rx channels.<br />
:o <code>multifreq_loopback.cpp</code> — This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>rfnoc_txrx_loopback.cpp</code> — This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>rfnoc_txrx_loopback_mem.cpp</code> — This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/examples/''] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> configuration file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes, and thus achieve higher data rates. This requires user configuration to <code>receivefunctions.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder before building the examples. Refer to <code>receivefunctions.cpp</code> for more information.<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run <code>rfnoc_txrx_loopback.cpp</code> by calling<br />
#: <code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of 2 samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to <code>samples_generator.py</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation.<br />
* Use the replay example to generate a waveform. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block'' Knowledge Base article] at [https://kb.ettus.com/Using_the_RFNoC_Replay_Block https://kb.ettus.com/Using_the_RFNoC_Replay_Block] for more information.<br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (<code>IQIQI...</code>). Refer to the <code>readDatFile.py</code> example in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for more information about viewing data files.<br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Open_Architecture_For_Radar_and_EW_Research#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
== Definitions ==<br />
<br />
''Warranted'' specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="50%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 3 MHz to 6 GHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 50px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 50px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 50px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 50px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
== Synchronization ==<br />
<br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew'' — The average phase offset between two channels. Described by how it varies over time, or from run to run.<br />
* ''Standard deviation of channel-to-channel skew'' — One number that describes the variation of phase offset between channels around the average skew over time, or from run to run.<br />
* ''Rx-Rx'' — Channel-to-channel specifically referring to Rx channels.<br />
<br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Open Architecture for Radar and EW Research. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
=== Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where µ<sub>n</sub> is average skew between two channels in measurement window ''n''<br />
:o µ''<sub>n</sub>'' – µ<sub>0</sub> is the average skew relative to t<sub>0</sub><br />
:o Range{µ<sub>0</sub>, µ<sub>1</sub>, … , µ<sub>N</sub>} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="50%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="50%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where µ<sub>nx</sub> is the average skew between channel ''n'' and reference channel in measurement window ''x''<br />
:o sd<sub>n</sub> = σ{µ<sub>n0</sub>, µ<sub>n1</sub>,…, µ<sub>n6</sub>} is the standard deviation of average skew for channel ''n'' using all measurement windows<br />
:o Average over 32 channels is µ{sd<sub>1</sub>…}<br />
:o Maximum of 32 channels is max{sd<sub>1</sub>…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
:o Where m<sub>nx</sub> is Rx ''n''-to-Rx 0 average skew in run ''x''<br />
:o sd<sub>n</sub> = σ{m<sub>n0</sub>, m<sub>n1</sub>, … m<sub>n9</sub>} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o r<sub>n</sub> = Max(Abs({m<sub>n0</sub>, m<sub>n1</sub>, … m<sub>n9</sub>} - m<sub>n0</sub>)) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{sd<sub>1</sub>, sd<sub>2</sub>,…,sd<sub>32</sub>} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{sd<sub>1</sub>, sd<sub>2</sub>,…,sd<sub>32</sub>} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{r<sub>1</sub>, r<sub>2</sub>,…,r<sub>32</sub>} is the system average range of Rx-Rx skew<br />
:o Maximum{r<sub>1</sub>, r<sub>2</sub>,…,r<sub>32</sub>} is the system maximum range of Rx-Rx skew<br />
<br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
| style="padding-left: 50px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
| style="padding-left: 50px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
| style="padding-left: 50px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
| style="padding-left: 50px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel. This was tested using a 32x32 MIMO loopback and sweeping carrier frequency from 1.0 GHz to 5.5 GHz.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
| style="padding-left: 50px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
| style="padding-left: 50px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
| style="padding-left: 50px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
| style="padding-left: 50px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
== Streaming ==<br />
<br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''] at [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide].<br />
<br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
= Appendix: Creating High-Channel-Count Systems =<br />
<br />
NI tested and configured the Open Architecture for Radar and EW Research for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
== Connection Considerations for High-Channel-Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
== LO Distribution for High-Channel-Count Systems ==<br />
<br />
Refer to [https://kb.ettus.com/Open_Architecture_For_Radar_and_EW_Research#Connecting_the_LO_Distribution ''Connecting the LO Distribution''] for information about LO distribution connections and best practices.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. 64x64 LO Sources and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
= Additional Resources =<br />
<br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Open Architecture for Radar and EW Research, software, and hardware components:<br />
<br />
* GitHub repository — Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Open Architecture for Radar and EW Research system. Visit [https://github.com/EttusResearch/refarch-multich https://github.com/EttusResearch/refarch-multich] .<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide'' — Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide].<br />
* ''N320/N321'' specifications — Lists specifications for the USRP N321 and USRP N320. Visit [https://kb.ettus.com/N320/N321 https://kb.ettus.com/N320/N321].<br />
* ''USRP Hardware Driver and USRP Manual — ''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit [https://files.ettus.com/manual/index.html https://files.ettus.com/manual/index.html].<br />
* ''USRP N320/N321 LO Distribution'' — Describes LO distribution connections and commands. Visit [https://kb.ettus.com/USRP_N320/N321_LO_Distribution https://kb.ettus.com/USRP_N320/N321_LO_Distribution].<br />
<br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications — Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base — Visit [https://kb.ettus.com/Knowledge_Base https://kb.ettus.com/Knowledge_Base].<br />
* SDR-Academy Informal Getting Started Videos — Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020 — Visit [https://kb.ettus.com/Training_Workshops https://kb.ettus.com/Training_Workshops] to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /><br />
<br />
[[Category:Application Notes]]</div>JovianWysockihttps://kb.ettus.com/index.php?title=Knowledge_Base&diff=5375Knowledge Base2022-04-20T13:44:19Z<p>JovianWysocki: /* Software Resources */</p>
<hr />
<div>Welcome to the Ettus Research Knowledge Base (KB). The KB is continuously being updated and expanded. If you have any suggestions, or do not find what you are looking for, then please [http://www.ettus.com/contact Contact Us].<br />
__NOTOC__<br />
<div class="row"><br />
<div class="col-1-3"><br />
== [[Getting Started Guides|<i class="fa fa-road"></i> Getting Started Guides]] ==<br />
<br />
'''Motherboards'''<br />
* [[B200/B210/B200mini/B205mini Getting Started Guides|B200/B210/B200mini/B205mini]]<br />
* [[Ettus USRP E300 Embedded Family Getting Started Guides|E310/E312/E313]]<br />
* [[E320 Getting Started Guide|E320]]<br />
* [[N200/N210 Getting Started Guides|N200/N210]]<br />
* [[USRP N300/N310/N320/N321 Getting Started Guide|N300/N310/N320/N321]]<br />
* [[X300/X310 Getting Started Guides|X300/X310]]<br />
* [[USRP-2974 Getting Started Guide|USRP-2974]]<br />
* [[USRP X410 Getting Started Guide|X410]]<br />
<br />
'''Daughterboards'''<br />
* [[BasicTX/BasicRX Getting Started Guides|BasicTX/BasicRX]]<br />
* [[CBX Getting Started Guides|CBX]]<br />
* [[LFTX/LFRX Getting Started Guides|LFTX/LFRX]]<br />
* [[SBX Getting Started Guides|SBX]]<br />
* [[TwinRX Getting Started Guides|TwinRX]]<br />
* [[UBX Getting Started Guides|UBX]]<br />
* [[WBX Getting Started Guides|WBX]]<br />
<br />
'''Other'''<br />
* [[Getting_Started_with_RFNoC_in_UHD_4.0|RFNoC Development (UHD 4.x)]]<br />
* [[RFNoC_4_Migration_Guide|RFNoC Migration Guide (UHD 3.x to UHD 4.x)]]<br />
* [[Getting_Started_with_RFNoC_Development|RFNoC Development (UHD 3.x)]]<br />
* [[Live SDR Environment Getting Started Guides|Live SDR Environment]]<br />
* [[OctoClock CDA-2990 Getting Started Guides|OctoClock CDA-2990]]<br />
* [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices|White Rabbit]]<br />
* [[Getting Started with DPDK and UHD|DPDK]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Hardware Resources|<i class="fa fa-cogs"></i> Hardware Resources]] ==<br />
'''Motherboards'''<br />
* [[B200/B210/B200mini/B205mini]]<br />
* [[Ettus USRP E300 Embedded Family Hardware Resources|E310/E312/E313]]<br />
* [[E320|E320]]<br />
* [[N200/N210]]<br />
* [[N300/N310]]<br />
* [[N320/N321]]<br />
* [[X300/X310]]<br />
* [[USRP-2974]]<br />
<br />
'''Daughterboards'''<br />
* [[BasicTX/BasicRX]]<br />
* [[CBX]]<br />
* [[LFTX/LFRX]]<br />
* [[SBX]]<br />
* [[TwinRX]]<br />
* [[UBX]]<br />
* [[WBX]]<br />
<br />
'''Other'''<br />
* [[OctoClock CDA-2990]]<br />
* [[GPSDO]]<br />
* [[Antennas]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Software Resources|<i class="fa fa-desktop"></i> Software Resources]] ==<br />
'''Ettus Products'''<br />
* [[UHD]]<br />
* [[UHD Python API]]<br />
* [[Getting_Started_with_RFNoC_in_UHD_4.0|RFNoC (UHD 4.x)]]<br />
* [[RFNoC]] (UHD 3.x)<br />
<br />
'''Third Party'''<br />
* [[GNU Radio]]<br />
* [[LabVIEW]]<br />
* [[Matlab/Simulink]]<br />
* [[OpenBTS]]<br />
* [[Eurecom OpenAirInterface (OAI)]]<br />
* [[srsLTE/srsUE]]<br />
* [[Gqrx]]<br />
* [[Fosphor]]<br />
<br />
'''Reference Architectures'''<br />
* [[Multichannel RF Reference Architecture]]<br />
<br />
</div><br />
</div><br />
<br />
<div class="row"><br />
<div class="col-1-3"><br />
<br />
== [[UHD and USRP User Manual|<i class="fa fa-flag"></i> UHD and USRP User Manual]] ==<br />
'''Software'''<br />
* [http://files.ettus.com/manual/ UHD Manual (master)]<br />
* [https://files.ettus.com/manual_archive/ UHD Manual Archive (previous releases)]<br />
<br />
'''Motherboards'''<br />
* [http://files.ettus.com/manual/page_usrp_b200.html B200/B210/B200mini/B205mini]<br />
* [http://files.ettus.com/manual/page_usrp_x3x0.html X300/X310]<br />
* [http://files.ettus.com/manual/page_usrp2.html N200/N210]<br />
* [http://files.ettus.com/manual/page_usrp_n3xx.html N300/N310/N320/N321]<br />
* [http://files.ettus.com/manual/page_usrp_e3xx.html E310/E312/E313/E320]<br />
<br />
'''Daughterboards'''<br />
* [http://files.ettus.com/manual/page_dboards.html#dboards_basictx BasicRX/LFRX]<br />
* [http://files.ettus.com/manual/page_dboards.html#dboards_basicrx BasicTX/LFTX]<br />
* [http://files.ettus.com/manual/page_dboards.html#dboards_cbx CBX]<br />
* [http://files.ettus.com/manual/page_dboards.html#dboards_sbx SBX]<br />
* [http://files.ettus.com/manual/page_dboards.html#dboards_wbx WBX]<br />
* [http://files.ettus.com/manual/page_dboards.html#dboards_ubx UBX]<br />
* [http://files.ettus.com/manual/page_dboards.html#dboards_twinrx TwinRX]<br />
<br />
'''Other'''<br />
* [http://files.ettus.com/manual/page_octoclock.html OctoClock]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Application Notes|<i class="fa fa-file-text-o"></i> Application Notes]] ==<br />
Application Notes (AN) and technical articles written by engineers, for engineers. These articles offer experienced analysis, design ideas, reference designs, and tutorials—to make you productive and successful using USRP devices.<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Additional Resources|<i class="fa fa-book"></i> Additional Resources]] ==<br />
* [[Suggested Reading|Suggested Reading]]<br />
* [[Suggested Videos|Suggested Videos]]<br />
* [[SDR Events]]<br />
* [[CGRAN]]<br />
</div><br />
</div><br />
<br />
<br />
<div class="row"><br />
<div class="col-1-3"><br />
<br />
== [[Technical Support|<i class="fa fa-life-ring"></i> Technical Support]] ==<br />
* [[Email|Email]]<br />
* [[Mailing Lists|Mailing Lists]]<br />
* [[Slack|Slack]]<br />
* [[Internet Relay Chat (IRC)|Internet Relay Chat (IRC)]]<br />
* [[StackExchange|StackExchange]]<br />
* [[Ordering and Fulfillment Help | Ordering and Fulfillment Help]]<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Faq|<i class="fa fa-info-circle"></i> FAQ]] ==<br />
* [[Technical FAQ|Technical]]<br />
* [[Licensing FAQ|Licensing]]<br />
<br />
<br />
</div><br />
<div class="col-1-3"><br />
<br />
== [[Legacy Products| <i class="fa fa-hourglass-end"></i> Legacy Products]] ==<br />
'''Motherboards'''<br />
* [[USRP1|USRP1]]<br />
* [[USRP2|USRP2]]<br />
* [[E100/E110|E100/E110]]<br />
* [[B100]]<br />
<br />
'''Daughterboards'''<br />
* [[DBSRX2]]<br />
* [[TVRX2]]<br />
* [[XCVR2450]]</div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5374Multichannel RF Reference Architecture2022-04-18T21:06:41Z<p>JovianWysocki: /* Synchronization */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5373Multichannel RF Reference Architecture2022-04-18T21:05:34Z<p>JovianWysocki: /* Resetting Configuration between Runs */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5372Multichannel RF Reference Architecture2022-04-18T21:05:18Z<p>JovianWysocki: /* Resetting Session between Runs */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5371Multichannel RF Reference Architecture2022-04-18T21:04:44Z<p>JovianWysocki: /* Resetting Session between Runs */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=File:AN822-HW_Overview.png&diff=5367File:AN822-HW Overview.png2022-04-18T20:01:07Z<p>JovianWysocki: JovianWysocki uploaded a new version of File:AN822-HW Overview.png</p>
<hr />
<div></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5366Multichannel RF Reference Architecture2022-04-18T19:47:22Z<p>JovianWysocki: /* Connecting the LO Distribution */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5365Multichannel RF Reference Architecture2022-04-18T19:45:50Z<p>JovianWysocki: /* Appendix: Creating High‑Channel‑Count Systems */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. 64x64 LO Sources and Tiers<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO IN cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 28. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 9. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5364Multichannel RF Reference Architecture2022-04-18T19:44:07Z<p>JovianWysocki: /* Streaming */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. Streaming to Disk Data Transfer Rate<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. Streaming Simultaneously to/from Disk Rate <br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5362Multichannel RF Reference Architecture2022-04-18T19:17:18Z<p>JovianWysocki: /* Resetting Session between Runs */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 4 channels<br />
| 0.012°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 4 channels<br />
| 0.039°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
<br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5361Multichannel RF Reference Architecture2022-04-18T19:15:47Z<p>JovianWysocki: /* Tx Phase Coherence One-Hour Stability */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
:o Transmitters: Stimulus simultaneously transmitted from any two channels<br />
:o Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
:o Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
:o Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
:o Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
:o Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5360Multichannel RF Reference Architecture2022-04-18T19:11:18Z<p>JovianWysocki: /* Resetting Configuration between RunsA USRP configuration is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 G...</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;" | Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.025°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.067°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5359Multichannel RF Reference Architecture2022-04-18T19:10:17Z<p>JovianWysocki: /* Resetting Session between RunsA USRP session is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run. */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!colspan="2" style="text-align:left;"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
|-<br />
|style="padding-left: 36px" | Average over 32 channels<br />
| 0.021°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
|style="padding-left: 36px" | Average of 32 channels<br />
| 0.070°<br />
|-<br />
|style="padding-left: 36px" | Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
<br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5358Multichannel RF Reference Architecture2022-04-18T19:08:25Z<p>JovianWysocki: /* Repeatability of Rx-Rx Phase Coherence */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
:o ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
:o ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'',..., ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
:o µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
:o Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',...,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
:o µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
:o Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',...,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
<br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5357Multichannel RF Reference Architecture2022-04-18T19:05:30Z<p>JovianWysocki: /* Short-Term Rx-Rx Phase CoherenceShort-term describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs). */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
:o Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
:o ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',..., ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
:o Average over 32 channels is µ{''sd<sub>1</sub>''...}<br />
:o Maximum of 32 channels is max{''sd<sub>1</sub>''...}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
<br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5356Multichannel RF Reference Architecture2022-04-18T19:04:11Z<p>JovianWysocki: /* Rx Phase Coherence One-Hour StabilityStability reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration. */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
:o Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
:o System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
:o Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
:o ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5355Multichannel RF Reference Architecture2022-04-18T19:03:31Z<p>JovianWysocki: /* Rx Phase Coherence One-Hour StabilityStability reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration. */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
o: Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
o: System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
o: Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
o: ''µ<sub>n</sub>'' - ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', ... , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
<br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5354Multichannel RF Reference Architecture2022-04-18T19:00:43Z<p>JovianWysocki: /* System Characteristics */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
|style="padding-left: 36px" | On USRP<br />
| FPGA<br />
|-<br />
|style="padding-left: 36px" | On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
|style="padding-left: 36px" | Format<br />
| Binary<br />
|-<br />
|style="padding-left: 36px" | Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
<br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5353Multichannel RF Reference Architecture2022-04-18T18:46:07Z<p>JovianWysocki: /* Measured Performance */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5352Multichannel RF Reference Architecture2022-04-18T18:45:29Z<p>JovianWysocki: /* Viewing Data Files */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based <code>readDatFile.py</code> example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/dat_analysis] folder. Refer to the comments in <code>readDatFile.py</code> for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5351Multichannel RF Reference Architecture2022-04-18T18:44:37Z<p>JovianWysocki: /* Generating Waveform Data Files */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in <code>replaycontrol.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
<br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5350Multichannel RF Reference Architecture2022-04-18T18:43:06Z<p>JovianWysocki: /* Configuration Files */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example <code>runconfig.cfg</code> file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the <code>rx-file-location</code> variable in the <code>runconfig.cfg</code> file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the <code>runconfig.cfg</code> example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
<br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5349Multichannel RF Reference Architecture2022-04-18T17:13:23Z<p>JovianWysocki: /* Building Examples */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, <code>setup_script.sh</code>.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/build/''] to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
: <code>cmake ..</code><br />
: <code>make –j</code><br /><br />
If a custom example needs to be compiled, you must modify [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt] to be used with <code>cmake</code>. To build a new example titled <code>new_example.cpp</code>, complete the following steps.<br />
<br />
# Add the following lines to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/CMakeLists.txt]:<br />
<br />
: <code>add_executable(new_example ''{relative File Location}''/new_example.cpp)</code><br />
<br />
: <code>message(STATUS &quot;New Example&quot;)</code><br />
<br />
: <code>target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/''build/.]</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
<br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5348Multichannel RF Reference Architecture2022-04-18T17:01:24Z<p>JovianWysocki: /* Example Source Code */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the <code>uhd/host/examples</code> directory:<br />
:o <code>rfnoc_radio_loopback.cpp</code>—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o <code>rfnoc_replay_samples_from_file.cpp</code>—This example uses the replay block to replay data from a file.<br />
:o <code>rfnoc_rx_to_file.cpp</code>— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o <code>Arch_iterative_loopback.cpp</code>—This example cycles through each Tx channel to all Rx channels.<br />
:o <code>Arch_multifreq_loopback.cpp</code>—This example cycles through different frequencies and calls <code>rfnoc_txrx_loopback.cpp</code>.<br />
:o <code>Arch_rfnoc_txrx_loopback.cpp</code>—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o <code>Arch_rfnoc_txrx_loopback_mem.cpp</code>—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o <code>Arch_rx_to_mem.cpp</code>—This example demonstrates receiving on all channels to memory.<br />
:o <code>Arch_txrx_fullduplex.cpp</code>—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o <code>Arch_dynamic_tx.cpp</code>—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
<br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5347Multichannel RF Reference Architecture2022-04-18T16:59:47Z<p>JovianWysocki: /* Optimizing the Host System */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
: <code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
: where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to <code>nicrb.sh</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the <code>.yaml</code> file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5346Multichannel RF Reference Architecture2022-04-18T16:35:56Z<p>JovianWysocki: /* Setting Up Additional USRPs */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
:o ''Updating the Linux File System''<br />
:o ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
:o Updating the FPGA Image<br />
* Update the <code>.yaml</code> file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
<br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5345Multichannel RF Reference Architecture2022-04-18T16:31:35Z<p>JovianWysocki: /* Setting Up the First USRP */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the <code>uhd_find_devices</code> example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to <code>''&lt;REFARCH&gt;''/config/usrp_filesystem.sh</code> in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to <code>9000</code>. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, <code>192.168.''XXX''.2</code>. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
<code>uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;</code><br /><br />
where <code>''&lt;N3xx_IP_ADDR&gt;''</code> is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command <code>uhd_find_devices</code>.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template <code>.yaml</code> file. Each host port and the USRP to which it is paired must have a matching subnet. After the <code>.yaml</code> file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5343Multichannel RF Reference Architecture2022-04-18T15:55:45Z<p>JovianWysocki: /* Installing and Configuring the Software */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The <code>refarch-multich</code> repository contains the following folders:<br />
<br />
* <code>config</code>—Contains script files that assist with the software installation process.<br />
* <code>docs</code>—Contains system bill of materials and reference architecture template.<br />
* <code>examples</code>—Contains reference architecture examples and configuration files.<br />
* <code>lib</code>—Contains the library source files.<br />
* <code>tools</code>—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5342Multichannel RF Reference Architecture2022-04-18T15:54:31Z<p>JovianWysocki: /* Connecting the LO Distribution */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to'' <code>sync.cpp</code> in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
<br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The ''refarch-multich'' repository contains the following folders:<br />
<br />
* ''config''—Contains script files that assist with the software installation process.<br />
* ''docs''—Contains system bill of materials and reference architecture template.<br />
* ''examples''—Contains reference architecture examples and configuration files.<br />
* ''lib''—Contains the library source files.<br />
* ''tools''—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5341Multichannel RF Reference Architecture2022-04-18T15:44:30Z<p>JovianWysocki: /* Installing and Configuring the Software */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to sync.cpp'' in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level ''refarch-multich'' directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The ''refarch-multich'' repository contains the following folders:<br />
<br />
* ''config''—Contains script files that assist with the software installation process.<br />
* ''docs''—Contains system bill of materials and reference architecture template.<br />
* ''examples''—Contains reference architecture examples and configuration files.<br />
* ''lib''—Contains the library source files.<br />
* ''tools''—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5340Multichannel RF Reference Architecture2022-04-18T14:30:58Z<p>JovianWysocki: /* Cabling the Hardware */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note''' Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in the following figure. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO IN. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''[https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Synchronization Synchronization]'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. ''Refer to sync.cpp'' in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 1. LO Source and Tiers<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 2. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 3. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 4. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config—Contains script files that assist with the software installation process.<br />
* docs—Contains system bill of materials and reference architecture template.<br />
* examples—Contains reference architecture examples and configuration files.<br />
* lib—Contains the library source files.<br />
* tools—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5339Multichannel RF Reference Architecture2022-04-18T14:03:35Z<p>JovianWysocki: /* Cabling the Hardware */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note'''Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in Figure 8. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to sync.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config—Contains script files that assist with the software installation process.<br />
* docs—Contains system bill of materials and reference architecture template.<br />
* examples—Contains reference architecture examples and configuration files.<br />
* lib—Contains the library source files.<br />
* tools—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5338Multichannel RF Reference Architecture2022-04-18T14:02:46Z<p>JovianWysocki: /* Software Overview */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Replay Block Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note'''Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in Figure 8. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to sync.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config—Contains script files that assist with the software installation process.<br />
* docs—Contains system bill of materials and reference architecture template.<br />
* examples—Contains reference architecture examples and configuration files.<br />
* lib—Contains the library source files.<br />
* tools—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5337Multichannel RF Reference Architecture2022-04-18T13:50:05Z<p>JovianWysocki: /* Architecture Overview */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
:o Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note'''Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in Figure 8. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to sync.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config—Contains script files that assist with the software installation process.<br />
* docs—Contains system bill of materials and reference architecture template.<br />
* examples—Contains reference architecture examples and configuration files.<br />
* lib—Contains the library source files.<br />
* tools—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5332Multichannel RF Reference Architecture2022-04-15T13:52:36Z<p>JovianWysocki: /* Viewing Data Files */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
* Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
* Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note'''Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in Figure 8. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to sync.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config—Contains script files that assist with the software installation process.<br />
* docs—Contains system bill of materials and reference architecture template.<br />
* examples—Contains reference architecture examples and configuration files.<br />
* lib—Contains the library source files.<br />
* tools—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5331Multichannel RF Reference Architecture2022-04-15T08:22:58Z<p>JovianWysocki: /* Tx Phase Coherence One-Hour Stability */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
* Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
* Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note'''Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in Figure 8. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to sync.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config—Contains script files that assist with the software installation process.<br />
* docs—Contains system bill of materials and reference architecture template.<br />
* examples—Contains reference architecture examples and configuration files.<br />
* lib—Contains the library source files.<br />
* tools—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich/tools/oarer_dat_analysis ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png&diff=5330File:AN822-Tx Phase Coherence Stability Graph System.png2022-04-15T08:22:11Z<p>JovianWysocki: </p>
<hr />
<div></div>JovianWysockihttps://kb.ettus.com/index.php?title=File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png&diff=5329File:AN822-Tx Phase Coherence Stability Graph Device.png2022-04-15T08:21:50Z<p>JovianWysocki: </p>
<hr />
<div></div>JovianWysockihttps://kb.ettus.com/index.php?title=File:AN822-MIMO_Loopback_32x32.png&diff=5328File:AN822-MIMO Loopback 32x32.png2022-04-15T08:15:30Z<p>JovianWysocki: JovianWysocki uploaded a new version of File:AN822-MIMO Loopback 32x32.png</p>
<hr />
<div></div>JovianWysockihttps://kb.ettus.com/index.php?title=File:AN822-MIMO_Loopback_16x16.png&diff=5327File:AN822-MIMO Loopback 16x16.png2022-04-15T08:15:03Z<p>JovianWysocki: JovianWysocki uploaded a new version of File:AN822-MIMO Loopback 16x16.png</p>
<hr />
<div></div>JovianWysockihttps://kb.ettus.com/index.php?title=File:AN822-MIMO_Loopback_8x8.png&diff=5326File:AN822-MIMO Loopback 8x8.png2022-04-15T08:14:00Z<p>JovianWysocki: JovianWysocki uploaded a new version of File:AN822-MIMO Loopback 8x8.png</p>
<hr />
<div></div>JovianWysockihttps://kb.ettus.com/index.php?title=File:AN822-MIMO_Loopback_8x8.png&diff=5325File:AN822-MIMO Loopback 8x8.png2022-04-15T08:12:35Z<p>JovianWysocki: JovianWysocki uploaded a new version of File:AN822-MIMO Loopback 8x8.png</p>
<hr />
<div></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5324Multichannel RF Reference Architecture2022-04-15T08:11:47Z<p>JovianWysocki: /* Building the System */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
* Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
* Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note'''Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in Figure 8. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to sync.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config—Contains script files that assist with the software installation process.<br />
* docs—Contains system bill of materials and reference architecture template.<br />
* examples—Contains reference architecture examples and configuration files.<br />
* lib—Contains the library source files.<br />
* tools—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich/tools/oarer_dat_analysis ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
<br />
[[File:media/image35.png|612x225px]]<br />
<br />
{|<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
<br />
[[File:media/image36.png|615x226px]]<br />
<br />
{|<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysockihttps://kb.ettus.com/index.php?title=File:AN822-Back_View.jpg&diff=5323File:AN822-Back View.jpg2022-04-15T08:10:13Z<p>JovianWysocki: </p>
<hr />
<div></div>JovianWysockihttps://kb.ettus.com/index.php?title=Multichannel_RF_Reference_Architecture&diff=5322Multichannel RF Reference Architecture2022-04-15T08:05:53Z<p>JovianWysocki: /* Building the System */</p>
<hr />
<div>== Application Note Number and Authors ==<br />
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki<br />
<!-- Internal use only: please do keep this updated!<br />
== Revision History ==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2021-10<br />
|style="text-align:center;"| Michael Dickens <br />
|style="text-align:center;"| Initial creation<br />
|} --><br />
<br />
<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. --><br />
<br />
This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.<br />
<br />
This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.<br />
<br />
<br />
= Architecture Overview =<br />
<br />
The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 1. System Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]<br />
|}<br />
<br />
The architecture features the following components:<br />
<br />
* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.<br />
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.<br />
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.<br />
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.<br />
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:<br />
* Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels<br />
* Bidirectional streaming of IQ data between 32 channels and server<br />
* Bill of materials for systems with up to 32 channels.<br />
<br />
The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 2. Reference Architecture Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]<br />
|}<br />
<br />
<span id="hardware-overview"></span><br />
== Hardware Overview ==<br />
<br />
The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.<br />
<br />
The following diagram provides an overview of the topography of the connections and primary hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 3. Hardware Overview<br />
|-<br />
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]<br />
|}<br />
<br />
For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br /><br />
https://github.com/EttusResearch/refarch-multich<br />
<br />
<span id="required-niettus-research-hardware"></span><br />
=== Required NI/Ettus Research Hardware ===<br />
<br />
This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:<br />
<br />
* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.<br />
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.<br />
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).<br />
<br />
<span id="validated-hardware"></span><br />
=== Validated Hardware ===<br />
<br />
Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:<br />
<br />
* Server<br />
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.<br />
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.<br />
* Network Interface Cards (NIC)<br />
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.<br />
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.<br />
* Storage<br />
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.<br />
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.<br />
* Power Splitter/Combiners<br />
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.<br />
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.<br />
* Cabling<br />
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).<br />
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.<br />
* Rack Accessories<br />
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.<br />
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.<br />
<br />
<span id="software-overview"></span><br />
== Software Overview ==<br />
<br />
The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).<br />
<br />
The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 4. Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 5. Stream from Host Data Movement<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]<br />
|}<br />
<br />
<span id="building-the-system"></span><br />
<br />
= Building the System =<br />
<br />
The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 6. Multichannel RF Reference Architecture System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 7. Rear View of System with Cabling<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Back_View.tiff|235x451px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 8. Rack Layout for 32x32 System<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]<br />
|}<br />
<br />
<span id="system-assembly-best-practices"></span><br />
== System Assembly Best Practices ==<br />
<br />
Adhere to the following best practices when installing, setting up, and connecting components to your system:<br />
<br />
* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.<br />
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.<br />
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.<br />
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.<br />
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.<br />
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].<br />
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']<br />
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].<br />
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.<br />
* Periodically check all cabling to ensure a secure connection.<br />
<br />
<span id="cabling-the-hardware"></span><br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
== Cabling the Hardware ==<br />
<br />
The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.<br />
<br />
:: '''Note'''Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).<br />
<br />
<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span><br />
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===<br />
<br />
You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).<br />
<br />
:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.<br />
<br />
:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.<br />
<br />
The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in Figure 8. All cables connecting between tiers must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 9. MIMO Loopback (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]<br />
|}<br />
<br />
The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 10. MIMO Loopback (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]<br />
|}<br />
<br />
The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 11. MIMO Loopback (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]<br />
|}<br />
<br />
<br />
<span id="connecting-10-mhz-and-pps"></span><br />
=== Connecting 10 MHz and PPS ===<br />
<br />
Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.<br />
<br />
The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 12. 10 MHz and PPS Connections (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]<br />
|}<br />
<br />
The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 13. 10 MHz and PPS Connections (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]<br />
|}<br />
<br />
The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 14. 10 MHz and PPS Connections (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]<br />
|}<br />
<br />
Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 15. 10 MHz and PPS Connections (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]<br />
|}<br />
<br />
<span id="connecting-the-lo-distribution"></span><br />
=== Connecting the LO Distribution ===<br />
<br />
Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.<br />
<br />
While connecting the LO distribution, adhere to the following best practices:<br />
<br />
<ul><br />
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li><br />
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to sync.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li><br />
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li><br />
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li><br />
<li><p>Use minimal identical cable length between tiers.</p><br />
<p>The following table lists the tiers for all system configurations.</p></li></ul><br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 5. Streaming to Memory Data Transfer Rate<br />
!width="25%"| System<br />
!width="25%"| Tier(s)<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 2x2<br />
| LO Source (0)<br />
| 1<br />
| —<br />
|-<br />
| 4x4<br />
| LO Source (0), 1<br />
| 1<br />
| 1<br />
|-<br />
| 8x8<br />
| LO Source (0), 1, 2<br />
| 2<br />
| 2<br />
|-<br />
| 16x16<br />
| LO Source (0), 1, 2<br />
| 3<br />
| 5<br />
|-<br />
| 32x32<br />
| LO Source (0), 1, 2, 3<br />
| 6<br />
| 10<br />
|}<br />
<br />
The following diagrams represent LO distribution signal connections.<br />
<br />
The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 16. LO Sharing (2x2)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]<br />
|}<br />
<br />
The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 17. LO Sharing (4x4)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]<br />
|}<br />
<br />
For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 18. LO Sharing (8x8)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 6. LO Sharing Cabling (8x8)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|}<br />
<br />
<br />
For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 19. LO Sharing (16x16)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]<br />
|}<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 7. LO Sharing Cabling (16x16)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier, TX LO OUT<br />
| 8<br />
|}<br />
<br />
<br />
For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 20. LO Sharing (32x32)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (32x32)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in System<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 2<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 16<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 16<br />
|}<br />
<br />
<span id="connecting-the-usrp-management-port"></span><br />
=== Connecting the USRP Management Port ===<br />
<br />
Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.<br />
<br />
:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.<br />
<br />
# Connect an RJ45 cable between the server and the switch.<br />
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.<br />
# (Optional) Connect a cable between the network switch and external internet for network management.<br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-the-sfp-ports"></span><br />
=== Connecting the SFP+ Ports ===<br />
<br />
The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.<br />
<br />
Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 21. Overview of Dataflow between the USRPs and the Server<br />
|-<br />
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]<br />
|}<br />
<br />
Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.<br />
<br />
Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:<br />
<br />
* Label the NICs and 10 GbE cables to identify each USRP connection.<br />
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.<br />
<br />
<span id="connecting-to-power"></span><br />
=== Connecting to Power ===<br />
<br />
Plug the USRPs, OctoClocks, server, and switch into the system PDU.<br />
<br />
<span id="installing-and-configuring-the-software"></span><br />
<br />
== Installing and Configuring the Software ==<br />
<br />
Complete the following steps to install the software and necessary dependencies on your host machine.<br />
<br />
# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.<br />
# Clone the repository to your computer using standard GitHub commands or download the repository directly.<br />
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:<br />
#: <code>chmod +x setup_script.sh</code><br />
# Run the script with the following command:<br /><br />
#: <code>sudo ./setup_script.sh</code><br />
<br />
The refarch-multich repository contains the following folders:<br />
<br />
* config—Contains script files that assist with the software installation process.<br />
* docs—Contains system bill of materials and reference architecture template.<br />
* examples—Contains reference architecture examples and configuration files.<br />
* lib—Contains the library source files.<br />
* tools—Contains Python-based data file generator and viewer.<br />
<br />
<span id="setting-up-the-first-usrp"></span><br />
<br />
== Setting Up the First USRP ==<br />
<br />
The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.<br />
<br />
::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.<br />
<br />
# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.<br />
# Follow the procedure in ''Updating the Linux File System''.<br />
<br />
:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.<br />
<br />
<ol start="3" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol><br />
<br />
:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.<br />
<br />
::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.<br />
<br />
<br />
<ol start="4" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br /><br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br /><br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol><br />
<br />
::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.<br />
<br />
<ol start="5" style="list-style-type: decimal;"><br />
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol><br />
<br />
::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br /><br />
::<code>$ sudo netplan apply</code><br />
<br />
<br />
<ol start="6" style="list-style-type: decimal;"><br />
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li><br />
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol><br />
<br />
<span id="setting-up-additional-usrps"></span><br />
<br />
== Setting Up Additional USRPs ==<br />
<br />
After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.<br />
<br />
* For subsequent devices, you may choose to execute the following procedures in parallel:<br />
* ''Updating the Linux File System''<br />
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.<br />
* Updating the FPGA Image<br />
* Update the .yaml file on the host computer after configuring each remaining USRP.<br />
<br />
<span id="optimizing-the-host-system"></span><br />
== Optimizing the Host System ==<br />
<br />
Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:<br />
<br />
* Increase the ring buffer for each interface by executing the following command in a terminal<br /><br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br /><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.<br />
<br />
:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.<br />
<br />
:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.<br />
<br />
* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.<br />
<br />
<span id="using-the-software"></span><br />
<br />
= Using the Software =<br />
<br />
<span id="example-source-code"></span><br />
== Example Source Code ==<br />
<br />
Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).<br />
<br />
* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:<br />
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.<br />
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.<br />
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.<br />
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:<br />
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.<br />
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.<br />
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.<br />
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.<br />
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.<br />
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.<br />
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.<br />
<br />
:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.<br />
<br />
<span id="building-examples"></span><br />
== Building Examples ==<br />
<br />
The reference architecture examples are compiled when you run the following setup script, setup_script.sh.<br />
<br />
If you need to make changes to the examples, you must recompile the code.<br />
<br />
# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.<br />
# Execute the following commands in the terminal:<br />
<br />
<code>cmake ..<br />
<br />
make –j<br />
</code><br />
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.<br />
<br />
# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:<br />
<br />
<code>add_executable(new_example {relative File Location}/new_example.cpp)<br />
<br />
message(STATUS &quot;New Example&quot;)<br />
<br />
target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)<br />
</code><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li><br />
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol><br />
<br />
<span id="configuration-files"></span><br />
== Configuration Files ==<br />
<br />
The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.<br />
<br />
* Refer to the example runconfig.cfg file for a detailed explanation of each argument.<br />
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.<br />
<br />
:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.<br />
<br />
<span id="running-the-examples"></span><br />
== Running the Examples ==<br />
<br />
# Open a terminal in the build folder.<br />
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br /><br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code><br />
<br />
<span id="generating-waveform-data-files"></span><br />
== Generating Waveform Data Files ==<br />
<br />
* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.<br />
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.<br />
<br />
<span id="viewing-data-files"></span><br />
== Viewing Data Files ==<br />
<br />
Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich/tools/oarer_dat_analysis ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.<br />
<br />
<span id="measured-performance"></span><br />
<br />
= Measured Performance =<br />
<br />
NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.<br />
<br />
<span id="definitions"></span><br />
== Definitions ==<br />
<br />
''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.<br />
<br />
''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.<br />
<br />
''Typical'' specifications describe the performance met by a majority of systems.<br />
<br />
''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.<br />
<br />
''Measured'' specifications describe the measured performance of a representative system.<br />
<br />
Specifications are ''Measured'' unless otherwise noted.<br />
<br />
<span id="system-characteristics"></span><br />
== System Characteristics ==<br />
<br />
{| class="wikitable"<br />
|width="49%"| Transmit channel count<br />
|width="50%"| Up to 32<br />
|-<br />
| Receive channel count<br />
| Up to 32<br />
|-<br />
| RF frequency coverage<br />
| 400 MHz<br />
|-<br />
| RF bandwidth<br />
| Up to 200 MHz instantaneous bandwidth per channel<br />
|-<br />
| Configuration plane<br />
| 1 GbE<br />
|-<br />
| Data plane<br />
| 10 GbE<br />
|-<br />
| Processing nodes<br />
|<br />
|-<br />
| On USRP<br />
| FPGA<br />
|-<br />
| On server<br />
| CPU<br />
|-<br />
| Data storage and streaming<br />
|<br />
|-<br />
| Format<br />
| Binary<br />
|-<br />
| Type<br />
| Complex interleaved U16<br />
|-<br />
| Acquisition mode<br />
| Streaming, finite records<br />
|}<br />
<br />
<span id="synchronization"></span><br />
== Synchronization ==<br />
<br />
<span id="synchronization-terminology"></span><br />
=== Synchronization Terminology ===<br />
<br />
In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.<br />
<br />
Phase coherence performance of the test system is described using the following metrics:<br />
<br />
* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.<br />
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.<br />
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.<br />
<br />
<span id="synchronization-methodology"></span><br />
=== Synchronization Methodology ===<br />
<br />
Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.<br />
<br />
<span id="rx-phase-coherence-one-hour-stability"></span><br />
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 2 GHz (S-band/L-band)<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)<br />
* Measurement window: 4 ms (2,500 samples)<br />
* Configuration:<br />
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP<br />
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 22. RF Configuration for Stability Measurements<br />
|-<br />
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]<br />
|}<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.139°<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
|-<br />
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable"<br />
|width="49%"| Range, system<br />
|width="50%"| 0.167°<br />
|}<br />
<br />
<span id="short-term-rx-rx-phase-coherence"></span><br />
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency<br />
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)<br />
* Analysis:<br />
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x<br />
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows<br />
* Average over 32 channels is µ{''sd<sub>1</sub>''…}<br />
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency<br />
|-<br />
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]<br />
|}<br />
<br />
<span id="repeatability-of-rx-rx-phase-coherence"></span><br />
=== Repeatability of Rx-Rx Phase Coherence ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 10 runs<br />
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx<br />
* Configured carrier frequency: 2 GHz<br />
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s<br />
* Measurement window: 8 µs (2,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew<br />
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew<br />
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew<br />
<br />
<span id="resetting-session-between-runs"></span><br />
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.021°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.145°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.070°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.483°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs"></span><br />
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====<br />
<br />
:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 32 channels<br />
| 0.025°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.066°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 32 channels<br />
| 0.067°<br />
|-<br />
| Maximum of 32 channels<br />
| 0.169°<br />
|}<br />
<br />
<span id="tx-phase-coherence-one-hour-stability"></span><br />
=== Tx Phase Coherence One-Hour Stability ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows<br />
* Measurement window: 35 ms (120,000 samples)<br />
* Configuration:<br />
* Transmitters: Stimulus simultaneously transmitted from any two channels<br />
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device<br />
* Analysis:<br />
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''<br />
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''<br />
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range<br />
<br />
Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub><br />
<br />
[[File:media/image35.png|612x225px]]<br />
<br />
{|<br />
|width="49%"| Range, single device<br />
|width="50%"| 0.128°<br />
|}<br />
<br />
Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub><br />
<br />
[[File:media/image36.png|615x226px]]<br />
<br />
{|<br />
|width="49%"| Range, system<br />
|width="50%"| 0.115°<br />
|}<br />
<br />
<span id="repeatability-of-tx-tx-phase-coherence"></span><br />
==== Repeatability of Tx-Tx Phase Coherence ====<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* 50 runs<br />
* Transmit from 4 Tx<br />
* Configured carrier frequency: 1 GHz<br />
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s<br />
* Measurement window: 35 µs (120,000: samples)<br />
* Analysis:<br />
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''<br />
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''<br />
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''<br />
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew<br />
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew<br />
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew<br />
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew<br />
<br />
<span id="resetting-session-between-runs-1"></span><br />
====== Resetting Session between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.012°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.039°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.089°<br />
|}<br />
<br />
<span id="resetting-configuration-between-runs-1"></span><br />
====== Resetting Configuration between Runs ======<br />
<br />
{| class="wikitable"<br />
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)<br />
!width="50%"|<br />
|-<br />
| Average over 4 channels<br />
| 0.014°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.020°<br />
|-<br />
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)<br />
|<br />
|-<br />
| Average of 4 channels<br />
| 0.041°<br />
|-<br />
| Maximum of 4 channels<br />
| 0.059°<br />
|}<br />
<br />
<span id="streaming"></span><br />
<br />
== Streaming ==<br />
<br />
<span id="streaming-to-memory-data-transfer-rate"></span><br />
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 122.88<br />
| 8<br />
| 10<br />
|}<br />
<br />
:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].<br />
<br />
<span id="streaming-to-disk-data-transfer-rate"></span><br />
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 33.33<br />
| 32<br />
| 10<br />
|-<br />
| 50<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|}<br />
<br />
<span id="streaming-simultaneously-tofrom-disk-rate"></span><br />
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===<br />
<br />
Measurements are valid under the following conditions:<br />
<br />
* Configured frequency: 2.5 GHz (S-band)<br />
* Configured stimulus: Single-tone<br />
* Number of channels: Varied<br />
* Duration: Single run, 10 s<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="33%"| Streaming Rate (MSample/s per Channel)<br />
!width="33%"| Number of Channels<br />
!width="33%"| Measurement Time (s)<br />
|-<br />
| 11.11<br />
| 32<br />
| 10<br />
|-<br />
| 25<br />
| 16<br />
| 10<br />
|-<br />
| 62.5<br />
| 8<br />
| 10<br />
|-<br />
| 62.5<br />
| 4<br />
| 10<br />
|}<br />
<br />
<span id="appendix-creating-highchannelcount-systems"></span><br />
<br />
= Appendix: Creating High‑Channel‑Count Systems =<br />
<br />
NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.<br />
<br />
<span id="connection-considerations-for-highchannelcount-systems"></span><br />
== Connection Considerations for High‑Channel‑Count Systems ==<br />
<br />
When the system channel count increases to 64 channels and greater, consider the following:<br />
<br />
* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.<br />
* Drift increases for typical measurements.<br />
<br />
<span id="lo-distribution-for-highchannelcount-systems"></span><br />
== LO Distribution for High‑Channel‑Count Systems ==<br />
<br />
Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.<br />
<br />
:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.<br />
<br />
The following table lists the tiers for the 64x64 system configuration.<br />
<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
!width="24%"| System<br />
!width="25%"| Tiers<br />
!width="25%"| USRP N321 Minimum Quantity<br />
!width="25%"| USRP N320 Maximum Quantity<br />
|-<br />
| 64x64<br />
| LO Source (0), 1, 2, 3<br />
| 11<br />
| 21<br />
|}<br />
<br />
The following diagram represents LO distribution signal connections.<br />
<br />
For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.<br />
<br />
{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"<br />
|+ Figure 23. LO Sharing (64x64)<br />
|-<br />
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]<br />
|}<br />
<br />
<br />
{| class="wikitable" style="text-align: center"<br />
|+ Table 8. LO Sharing Cabling (64x64)<br />
!width="33%"| Cable Color in Diagram<br />
!width="33%"| Cable Tier<br />
!width="33%"| Number of Cables in Tier<br />
|-<br />
| [[File:AN822-LO_Cable_table_black.png|13x13px]]<br />
| LO Source Tier 0<br />
| 4<br />
|-<br />
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]<br />
| Distribution Tier 1, RX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]<br />
| Distribution Tier 1, TX LO OUT<br />
| 8<br />
|-<br />
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]<br />
| Distribution Tier 2, RX LO OUT<br />
| 32<br />
|-<br />
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]<br />
| Distribution Tier 2, TX LO OUT<br />
| 32<br />
|}<br />
<br />
<span id="additional-resources"></span><br />
<br />
= Additional Resources =<br />
<br />
<span id="related-documentation"></span><br />
== Related Documentation ==<br />
<br />
Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:<br />
<br />
* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.<br />
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.<br />
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.<br />
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.<br />
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.<br />
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.<br />
<br />
<span id="training-resources"></span><br />
== Training Resources ==<br />
<br />
* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.<br />
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.<br />
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].<br />
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.<br />
<br />
= Footnotes =<br />
<references /></div>JovianWysocki