View source for Multichannel RF Reference Architecture

== Application Note Number and Authors ==
'''AN-822''' by Dylan Baros, Michael Dickens, Joseph Paller, Neel Pandeya, and Jovian Wysocki
<!-- Internal use only: please do keep this updated!
== Revision History ==
{| class="wikitable"
!Date
!Author
!Details
|-
|style="text-align:center;"| 2021-10
|style="text-align:center;"| Michael Dickens 
|style="text-align:center;"| Initial creation
|} -->

<!-- NOTE: DO NOT update this AppNote without the knowledge and approval of an NI technical write such as Susan Collier and a leader such as Abhay Samant. -->

This document provides guidance for designing a system that uses the NI Multichannel RF Reference Architecture version 1.1.

This document describes how to design a system that enables synchronization between Receive (Rx) channels, Transmit (Tx) channels, and Transmit-Receive (Tx-Rx) channels, as well as high-throughput data sample transport from the software-defined radio (SDR) to a server system over 10 GbE data links. This document provides a starting point in the creation of your specific system configuration. A basic understanding of Linux, command line, C++, Git, and networking is required to build and develop on the NI Multichannel RF Reference Architecture system.


= Architecture Overview =

The Multichannel RF Reference Architecture allows engineers to quickly scale from software simulation to hardware demonstration. It features direct conversion RF sampling (200 MHz instantaneous bandwidth) with NI USRPs for L-band, S-band, and C-band radar applications, enabling coherent operation through shared local oscillator (LO), PPS, and 10 MHz reference clock signals.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 1. System Overview
|-
| style="border-style: none;" | [[File:AN822-System_Overview.png|544x91px|center]]
|}

The architecture features the following components:

* Document, bill of materials, and interconnect diagrams that describe how to set up a system with up to 32 channels of Rx-Rx, Tx-Rx, and Tx-Tx synchronization.
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s to maintain phase coherence between multiple channels.
* Reference library that contains functions to configure and control the USRP N321s and USRP N320s for high-rate data transfer from multiple channels to memory or storage.
* C++ examples that demonstrate how to use reference library functions to configure and control a multichannel system.
* Documentation that lists installation and configuration information for a 32x32 system, as well as best practices for development and validation of the system. ''N''x''M'' refers to systems, where ''N'' is the number of transmit channels and ''M'' is the number of receive channels.
* Documentation that lists performance benchmarks with validated results for systems with up to 32 channels for the following specifications:
:o Synchronization between 32 Rx-Rx channels, 6 Tx-Tx channels, and 32 Tx-Rx channels
:o Bidirectional streaming of IQ data between 32 channels and server
* Bill of materials for systems with up to 32 channels.

The reference architecture demonstrates using RFNoC to transmit from both the replay block as well as the host.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 2. Reference Architecture Overview
|-
| style="border-style: none;" | [[File:AN822-Reference_Architecture_Overview.png|512x274px|center]]
|}

<span id="hardware-overview"></span>
== Hardware Overview ==

The reference architecture uses USRP N321s and USRP N320s that scale from 2 to 32 phase-coherent transmit and receive channels.

The following diagram provides an overview of the topography of the connections and primary hardware components.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 3. Hardware Overview
|-
| style="border-style: none;" | [[File:AN822-HW_Overview.png|565x361px|center]]
|}

For a complete list of required hardware for the Multichannel RF Reference Architecture, refer to the bill of materials in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder, where ''&lt;REFARCH&gt;'' is an alias for the following GitHub file folder location:<br />
https://github.com/EttusResearch/refarch-multich

<span id="required-niettus-research-hardware"></span>
=== Required NI/Ettus Research Hardware ===

This section describes the NI/Ettus Research products required for generating and acquiring RF signals as well as the synchronization signals—PPS, 10 MHz, and Local Oscillator (LO)—shown in Figure 3:

* '''USRP N321'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features a power splitter for LO distribution, 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.
* '''USRP N320'''—Networked software-defined radio with 2-channel RF front end and 250 MSample/s on both Tx and Rx. Features LO sharing and 10 MHz shared reference clock for coherent MIMO functionality, shared PPS for enabling shared triggering, dual 10 GbE ports for data transfer, and an RJ45 management port for device management.
* '''CDA-2990 with GPSDO'''—Clock distribution device called an OctoClock that generates and distributes the GPSDO signals—10 MHz reference (REF) and 1 Hz (PPS)—and features a management port.
* '''NI RF Torque Screwdriver and SMA Driver Bit''' (NI part number 780895-01)—A torque screwdriver is ''strongly'' recommended to securely tighten all connections to/from the USRPs in the system. You can also order the RF SMA Driver Bit Only (NI part number 780894-01).

<span id="validated-hardware"></span>
=== Validated Hardware ===

Testing and deploying a system requires hardware from both NI and third-party vendors. The following component options have been validated and are recommended for most Multichannel RF Reference Architecture systems:

* Server
:o '''Supermicro 4124GS-TNR'''—Server with dual AMD EPYC processors and 10 PCIe 4 x16 slots. Numerous PCIe lanes from the CPU support many 10 GbE connections and the PCIe processing required for high-channel-count systems.
:o '''Trenton Systems BAM3U'''—Server with Intel dual 3rd generation Xeon Ice Lake processors, 11 Gen 4.0 PCIe slots (5 x16 slots, 6 x8 slots), and 24x DDR4-3200 ECC RDIMM slots for high-channel-count systems.
* Network Interface Cards (NIC)
:o '''Intel X710-DA4'''—Network card that supports x4 10 GbE connections.
:o '''Intel E810-CQDA2'''—Network card that supports two 100 GbE connections, or, with a breakout cable, x8 10 GbE connections.
* Storage
:o '''HighPoint SSD7505'''—PCIe carrier board that holds x4 M.2 drives and supports either a software or hardware RAID configuration.
:o '''Sabrent 2TB Rocket 4 PLUS'''—NVME M.2 SSD drive that supports consecutive writes rates of over 6 GB/s.
* Power Splitter/Combiners
:o '''Mini-Circuits 8-Way ZN8PD1-63W-S+'''—DC pass, 8-way -0°, 50 Ω, 500 MHz to 6,000 MHz power splitter/combiner.
:o '''Mini-Circuits 4-Way ZN4PD1-63HP-S+'''—High power, DC pass, 4-way -0°, 50 Ω, 30 W, 500 MHz to 6,000 MHz power splitter/combiner.
* Cabling
:o '''SMA Cable, Loop Back Cable Kit''' (NI part number 782781-01)—Loopback cable kit includes 2 SMA-M to SMA-M cables (60 cm/2 ft) and 2 SMA-F to SMA-M Attenuators (30 dB, 50 Ohm, DC-6 GHz).
:o '''SFP+ Cable, 10 Gigabit Ethernet Cable w/ SFP+ Terminations (1 Meter)''' (NI part number 783343-01)—SFP+ cable recommended for USRPs with 10 GigE interfaces. It can be used to connect the USRP to a 10 GigE switch or network adapter.
* Rack Accessories
:o '''Tripp Lite Rackmount Rack Extension Bracket''' (B018-000-1P5)—Bracket required to mount the USRPs in a rack.
:o '''Z-Bracket'''—Bracket required to mount the USRPs in a rack to meet environmental specifications.

<span id="software-overview"></span>
== Software Overview ==

The reference software, shown in Figure 2, consists of a C++ reference library and examples, using USRP Hardware Driver (UHD) version 4.1. The reference software is supported on Ubuntu Linux 20.04. It demonstrates how to assemble multi-USRP systems through the UHD RFNoC API and replay block. It contains a Python-based binary int16 data file (.dat) generator and viewer. The reference software is written to be scalable for systems 2x2 to 32x32 and beyond, as well as Tx or Rx only (2 to 32 channels and beyond).

The software provides examples that stream data from the replay block as well as the host, as shown in the following figures.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 4. Replay Block Data Movement
|-
| style="border-style: none;" | [[File:AN822-SW_Data_Movement.png|366x396px|center]]
|}

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 5. Stream from Host Data Movement
|-
| style="border-style: none;" | [[File:AN822-SW_Tx_Streaming_Data_Movement.png|366x396px|center]]
|}

<span id="building-the-system"></span>

= Building the System =

The following figures show a 32x32 system layout. NI recommends using a rack for this system to simplify connections between the different hardware components.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 6. Multichannel RF Reference Architecture System
|-
| style="border-style: none;" | [[File:AN822-OARER_three-quarter_view_with_code_on_screen.png|235x451px|center]]
|}

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 7. Rear View of System with Cabling
|-
| style="border-style: none;" | [[File:AN822-Back_View.jpg|235x451px|center]]
|}

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 8. Rack Layout for 32x32 System
|-
| style="border-style: none;" | [[File:AN822-Rack_Layout_32x32.png|235x451px|center]]
|}

<span id="system-assembly-best-practices"></span>
== System Assembly Best Practices ==

Adhere to the following best practices when installing, setting up, and connecting components to your system:

* Number and label the USRPs with serial numbers on the front of the devices. Visit [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/docs/32_Channel_Template_layout_BOM_and_wiring.ods''] for a customizable template.
* While running the system in loopback, attach 30 dB attenuators to all output channels to prevent damage to the USRPs.
* Arrange the equipment in the rack to optimize proper airflow. USRP N321s and USRP N320s facilitate airflow from right to left, and the server facilitates airflow from front to back. NI recommends using Z-brackets to vertically stagger installation of USRPs.
* Position the lowest USRP in the rack a few units up from the floor to allow for adequate device connector accessibility.
* NI recommends positioning the USRPs halfway to 2/3 toward the rear of the rack to allow easier bundling of cables.
* Label the NICs and 10 GbE cables to identify each USRP connection, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_SFP.2B_Ports ''Connecting the SFP+ Ports''].
* Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320 devices, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Connecting_the_LO_Distribution ''Connecting the LO Distribution'']
* Keep the end-to-end path of each Rx, Tx, LO, PPS, and 10 MHz signal identical in length, using the shortest cable possible per tier, as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture#Cabling_the_Hardware ''Cabling the Hardware''].
* Use a calibrated torque wrench to connect all SMA connections to industry standard torque measurements.
* Periodically check all cabling to ensure a secure connection.

<span id="cabling-the-hardware"></span>

== Cabling the Hardware ==

The following topics describe the connections for 8x8, 16x16, and 32x32 system connections.

:: '''Note'''Ensure that you use a calibrated torque wrench for all cabling connections. NI recommends the RF Torque Screwdriver and SMA Driver Bit (NI part number 780895-01).

<span id="optional-connecting-the-rf-rx-tx-ports-for-a-mimo-loopback"></span>
=== (Optional) Connecting the RF RX-TX Ports for a MIMO Loopback ===

You can use the MIMO loopback to validate the setup of the system and provide a baseline for synchronization calibration. Create the MIMO loopback by connecting the USRP TX/RX connectors (used for transmitting signals) to a combiner, connecting the combiner to a splitter, and connecting the splitter to the USRP RX2 connectors (used for receiving signals).

:: '''Notice''' 30 dB attenuators must be used on every TX to ensure safe use of the MIMO loopback. Using the system without attenuators will overload the RX and will permanently damage the USRPs.

:: '''Note''' NI recommends keeping all cables in the same tier the same length to reduce skew.

The 8x8 system requires one 8-way splitter and one 8-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way splitter and an 8-way combiner, as shown in Figure 8. All cables connecting between tiers must be the same length and type.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 9. MIMO Loopback (8x8)
|-
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_8x8.png|492x207px|center]]
|}

The 16x16 system requires two 8-way splitters, one 2-way splitter, two 8-way combiners, and one 2-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the two 8-way combiners to a 2-way combiner and the two 8-way splitters to a 2-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 10. MIMO Loopback (16x16)
|-
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_16x16.png|578x288px|center]]
|}

The 32x32 system requires four 8-way splitters, one 4-way splitter, four 8-way combiners, and one 4-way combiner. To set up the MIMO loopback, connect the RF 0/1 TX/RX and RX2 connectors on the USRP to an 8-way combiner and an 8-way splitter. Then connect the four 8-way combiners to a 4-way combiner and the four 8-way splitters to a 4-way splitter. All cables connecting between tiers, shown in the following figure, must be the same length and type.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 11. MIMO Loopback (32x32)
|-
| style="border-style: none;" | [[File:AN822-MIMO_Loopback_32x32.png|578x534px|center]]
|}


<span id="connecting-10-mhz-and-pps"></span>
=== Connecting 10 MHz and PPS ===

Connect a 10 MHz OUT connector on the OctoClock to the REF IN connector on the USRP. Connect a PPS OUT connector on the OctoClock to the PPS connector on the USRP.

The 4x4 system requires one OctoClock with GPSDO, two USRPs, and uses two 10 MHz and two PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 12. 10 MHz and PPS Connections (4x4)
|-
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_4x4.png|481x157px|center]]
|}

The 8x8 system requires one OctoClock with GPSDO, four USRPs, and uses four 10 MHz and four PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 13. 10 MHz and PPS Connections (8x8)
|-
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_8x8.png|481x157px|center]]
|}

The 16x16 system setup requires one OctoClock with GPSDO, eight USRPs, and uses eight 10 MHz and eight PPS OUT SMA connectors. Ensure that the source selection switch is set to internal on OctoClock 0 prior to power up.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 14. 10 MHz and PPS Connections (16x16)
|-
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_16x16.png|477x155px|center]]
|}

Scaling to a 32x32 system requires one OctoClock with GPSDO (OctoClock 0, shown in the following figure), two additional OctoClocks (OctoClocks 1 and 2), and 16 USRPs. The source 10 MHz and PPS signals originate from OctoClock 0, which drives the OctoClock 1 and 2 inputs. OctoClock 1 and 2 each distribute all eight output 10 MHz and PPS signals to USRPs, 16 in aggregate. All cables connected from OctoClock 0 must be the same length. All cables connected to the USRPs must be the same length. Ensure that the source selection switch is set to internal on OctoClock 0, and that the source selection switch is set to external on OctoClocks 1 and 2. The selection switches must be set prior to power up.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 15. 10 MHz and PPS Connections (32x32)
|-
| style="border-style: none;" | [[File:AN822-10MHz-PPS_Connections_32x32.png|607x177px|center]]
|}

<span id="connecting-the-lo-distribution"></span>
=== Connecting the LO Distribution ===

Connect the TX LO OUT, RX LO IN, and TX LO IN connectors on the USRPs for LO distribution.

:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to ''[https://kb.ettus.com/USRP_N320/N321_LO_Distribution USRP N320/N321 LO Distribution]'' for connection diagrams.

While connecting the LO distribution, adhere to the following best practices:

<ul>
<li><p>Tx and Rx must share the same LO connection. The LO sharing configuration for USRP N321s and USRP N320s in this system differs slightly from the LO sharing diagram on the Ettus Knowledge Base<ref><p>The [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] Knowledge Base article does not show one LO signal shared across Tx and Rx.</p></ref>. Distributing the LO from one channel to all other channels results in phase coherence across all channels, and there will be a constant non-zero phase offset between any two channels. This phase offset can be empirically measured and compensated for in software thus resulting in all channels being phase aligned. The overall system is phase synchronous under a wide range of conditions; refer to ''Synchronization'' for more information about measured synchronization performance.</p></li>
<li><p>Configure all USRPs, including the LO Source USRP, to use external LO inputs. Refer to sync.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/lib/] folder for information about how the reference software configures the LOs on the USRPs.</p></li>
<li><p>Use tiered LO signal distribution and identical cable lengths between any specific tiers.</p></li>
<li><p>Connect the RX LO OUT and TX LO OUT cabling from Distribution Tier of USRP N321s to the nearest USRP N320s, as shown in the following figures.</p></li>
<li><p>Use minimal identical cable length between tiers.</p>
<p>The following table lists the tiers for all system configurations.</p></li></ul>



{| class="wikitable" style="text-align: center"
|+ Table 5. Streaming to Memory Data Transfer Rate
!width="25%"| System
!width="25%"| Tier(s)
!width="25%"| USRP N321 Minimum Quantity
!width="25%"| USRP N320 Maximum Quantity
|-
| 2x2
| LO Source (0)
| 1
| —
|-
| 4x4
| LO Source (0), 1
| 1
| 1
|-
| 8x8
| LO Source (0), 1, 2
| 2
| 2
|-
| 16x16
| LO Source (0), 1, 2
| 3
| 5
|-
| 32x32
| LO Source (0), 1, 2, 3
| 6
| 10
|}

The following diagrams represent LO distribution signal connections.

The 2x2 system requires two cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 16. LO Sharing (2x2)
|-
| style="border-style: none;" | [[File:AN822-LO_Dist_2x2.png|384x71px|center]]
|}

The 4x4 system requires four cables of identical length. Note that you cannot combine LO signals. NI recommends that all LO distribution cables originating from the same tier are the same length.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 17. LO Sharing (4x4)
|-
| style="border-style: none;" | [[File:AN822-LO_Dist_4X4.png|382x144px|center]]
|}

For the 8x8 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. Note that you cannot combine LO signals.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 18. LO Sharing (8x8)
|-
| style="border-style: none;" | [[File:AN822-LO_Dist_8X8.png|469x234px|center]]
|}



{| class="wikitable" style="text-align: center"
|+ Table 6. LO Sharing Cabling (8x8)
!width="33%"| Cable Color in Diagram
!width="33%"| Cable Tier
!width="33%"| Number of Cables in Tier
|-
| [[File:AN822-LO_Cable_table_black.png|13x13px]]
| LO Source Tier 0
| 2
|-
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]
| Distribution Tier 1, RX LO OUT
| 4
|-
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]
| Distribution Tier 1, TX LO OUT
| 4
|}


For the 16x16 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from Tier 1 USRP N321s must be the same length. Note that you cannot combine LO signals.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 19. LO Sharing (16x16)
|-
| style="border-style: none;" | [[File:AN822-LO_Dist_16X16.png|614x211px|center]]
|}



{| class="wikitable" style="text-align: center"
|+ Table 7. LO Sharing Cabling (16x16)
!width="33%"| Cable Color in Diagram
!width="33%"| Cable Tier
!width="33%"| Number of Cables in Tier
|-
| [[File:AN822-LO_Cable_table_black.png|13x13px]]
| LO Source Tier 0
| 4
|-
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]
| Distribution Tier, RX LO OUT
| 8
|-
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]
| Distribution Tier, TX LO OUT
| 8
|}


For the 32x32 system, both RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.


{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 20. LO Sharing (32x32)
|-
| style="border-style: none;" | [[File:AN822-LO_Dist_32X32.png|607x326px]]
|}

{| class="wikitable" style="text-align: center"
|+ Table 8. LO Sharing Cabling (32x32)
!width="33%"| Cable Color in Diagram
!width="33%"| Cable Tier
!width="33%"| Number of Cables in System
|-
| [[File:AN822-LO_Cable_table_black.png|13x13px]]
| LO Source Tier 0
| 2
|-
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]
| Distribution Tier 1, RX LO OUT
| 4
|-
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]
| Distribution Tier 1, TX LO OUT
| 4
|-
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]
| Distribution Tier 2, RX LO OUT
| 16
|-
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]
| Distribution Tier 2, TX LO OUT
| 16
|}

<span id="connecting-the-usrp-management-port"></span>
=== Connecting the USRP Management Port ===

Make the following RJ45 connections to allow USRP file system configuration and updates later in the process.

:: '''Note''' The network port on the USRPs in this system use DHCP by default. You must either have a DHCP server running on an upstream networked device or set a static IP address for each USRP.

# Connect an RJ45 cable between the server and the switch.
# Connect an RJ45 cable between the Management Port on each USRP and a port on one of the switches.
# (Optional) Connect a cable between the network switch and external internet for network management.

:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.

<span id="connecting-the-sfp-ports"></span>
=== Connecting the SFP+ Ports ===

The server contains multiple network interface cards (NIC), each supporting multiple SFP+ ports.

Each USRP has two SFP+ ports, SFP+ 0 and SPF+ 1; however, the reference architecture assumes the use of SFP+ 0.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 21. Overview of Dataflow between the USRPs and the Server
|-
| style="border-style: none;" | [[File:AN822-SFP_Dataflow.png|528x291px|center]]
|}

Connect an SFP+ cable between the SFP+ 0 port on each USRP and a port on one of the NICs in the server.

Assign each USRP to a particular network address. NI recommends that you organize your network in the following ways:

* Label the NICs and 10 GbE cables to identify each USRP connection.
* Create a document for tracking USRP serial numbers with assigned network address. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/docs/] folder for a customizable template.

<span id="connecting-to-power"></span>
=== Connecting to Power ===

Plug the USRPs, OctoClocks, server, and switch into the system PDU.

<span id="installing-and-configuring-the-software"></span>

== Installing and Configuring the Software ==

Complete the following steps to install the software and necessary dependencies on your host machine.

# Navigate to the GitHub repository, located at https://github.com/EttusResearch/refarch-multich, to get the reference software.
# Clone the repository to your computer using standard GitHub commands or download the repository directly.
# In a terminal, navigate to the top-level refarch-multich directory and issue the following command to make the setup script executable:
#: <code>chmod +x setup_script.sh</code>
# Run the script with the following command:<br />
#: <code>sudo ./setup_script.sh</code>

The refarch-multich repository contains the following folders:

* config—Contains script files that assist with the software installation process.
* docs—Contains system bill of materials and reference architecture template.
* examples—Contains reference architecture examples and configuration files.
* lib—Contains the library source files.
* tools—Contains Python-based data file generator and viewer.

<span id="setting-up-the-first-usrp"></span>

== Setting Up the First USRP ==

The following procedure references the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide'']. Follow the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide] procedure with the specific steps below.

::'''Note''' When you run the scripts in this section, ''all'' USRPs connected to the host are updated. Run the uhd_find_devices example for a list of USRPs that are connected to the host.

# Follow the ''Connecting to the ARM via SSH'' procedure in ''Connecting the Device''. You do not need to set up the USRPs for a serial console connection.
# Follow the procedure in ''Updating the Linux File System''.

:: '''Note''' If the procedure in ''Updating the file system with Mender'' fails, you must follow the process in ''Updating the files system by writing the disk image''. For a script that updates all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_filesystem.sh in the GitHub repository.

<ol start="3" style="list-style-type: decimal;">
<li><p>Follow the procedure in ''Updating the Network Configurations''. Set the MTU to 9000. Every USRP SFP+ ports IP address must have a unique subnet. Change the subnet by replacing ''XXX'' in the address, for example, 192.168.''XXX''.2. If you make any changes to the network configurations, NI recommends that you restart the USRP for those changes to take effect.</p></li></ol>

:: '''Note''' NI recommends that you create a document for tracking USRP serial numbers with assigned network addresses. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]docs/ folder for a customizable template. For network addresses, NI recommends that you assign static IP addresses for the management port IPs. Refer to the procedure in ''Updating the Network Configurations'' to update eth0.network file on the USRP.

::For a script to update the SFP ports on all USRPs automatically, refer to [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/]config/usrp_ip_config.sh in the GitHub repository. The script changes the sfp1 port by default but can be modified to change the sfp0 port. You still need to manually configure the host as listed in step 5.


<ol start="4" style="list-style-type: decimal;">
<li><p>Follow the ''Network Mode FPGA Image Update'' procedure in ''Updating the FPGA Image'' to update to the XG image. Use the following command to configure both SFP ports to 10 Gb links:<br />
uhd_image_loader --args &quot;type=n3xx,addr''=&lt;N3xx_IP_ADDR&gt;'',fpga=XG&quot;<br />
where ''&lt;N3xx_IP_ADDR&gt;'' is the actual IP address of the USRP being updated. You can find this address by correlating the serial number on the device with the IP address returned by the command uhd_find_devices.</p></li></ol>

::'''Note''' For a script to update the FPGA images on all USRPs connected to the system, refer to ''&lt;REFARCH&gt;''/config/usrp_image_loader.sh on the GitHub repository.

<ol start="5" style="list-style-type: decimal;">
<li><p>Follow the ''Dual 10Gb Streaming SFP Ports 0/1'' procedure in ''Setting Up a Streaming Connection'' to configure the host Ethernet adapters for 10 GbE connection to the USRP.</p></li></ol>

::'''Note''' Ubuntu Linux uses Netplan to manage IP addresses of network cards, however, other operating systems may configure the network interfaces differently. Visit the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/config/''] folder to find a customizable template .yaml file. Each host port and the USRP to which it is paired must have a matching subnet. After the .yaml file is configured, copy it to the <code>/etc/netplan</code> directory—overwriting <code>01-network-manager-all.yaml</code>, if necessary. Then use the following command to apply the network configuration:<br />
::<code>$ sudo netplan apply</code>


<ol start="6" style="list-style-type: decimal;">
<li><p>Follow the procedure in ''Verifying Device Operation''.</p></li>
<li><p>(Optional) Follow the procedure in ''USRP N3xx Device Specific Operations'' to configure USRP settings.</p></li></ol>

<span id="setting-up-additional-usrps"></span>

== Setting Up Additional USRPs ==

After setting up the first USRP, follow the procedure in ''Setting Up the First USRP'' for the remaining USRPs with the specific details listed below.

* For subsequent devices, you may choose to execute the following procedures in parallel:
* ''Updating the Linux File System''
* ''Updating the Network Configurations''—Each USRP port must be configured to have a different subnet.
* Updating the FPGA Image
* Update the .yaml file on the host computer after configuring each remaining USRP.

<span id="optimizing-the-host-system"></span>
== Optimizing the Host System ==

Adjust certain settings to optimize the host system. Most of these settings are completed for you through the setup script. NI recommends that you manually configure the following optimizations:

* Increase the ring buffer for each interface by executing the following command in a terminal<br />
<code>sudo ethtool -G ''&lt;interface&gt;'' tx 4096 rx 4096</code><br />
where <code>''&lt;interface&gt;''</code> is the network interface of the specific USRP. You can find this interface through the Network Manager or by executing the following command in a terminal <code>ifconfig</code>.

:: '''Note''' Increasing the ring buffer of an interface results in more memory reserved. When setting the ring buffer, make sure to monitor your memory usage. A large number of interfaces can result in hundreds of GB of memory usage. You may need to restart the host system to clear the allocated memory.

:: '''Note''' This command is executed once for each interface being used. For example, a system with 16 USRPs and 16 interfaces runs 16 times and results in around 65 GB of memory usage. A script has been provided to assist with these settings. The user is required to insert the names of the host interfaces. Refer to nicrb.sh in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/config/] folder for more information. This script may need to be run after each host reboot. You can reference the interface names in the .yaml file configured in ''Setting Up the First USRP Device''.

* Enable hyperthreading on the server to improve multi-threaded performance. Refer to the server BIOS settings for information.

<span id="using-the-software"></span>

= Using the Software  =

<span id="example-source-code"></span>
== Example Source Code ==

Multichannel RF Reference Architecture is built using the RFNoC API that ships with the USRP Hardware Driver (UHD).

* '''UHD RFNoC Example Source Code'''— NI recommends that users who are new to UHD or the RFNoC API begin with the following UHD RFNoC examples, located in the uhd/host/examples directory:
:o rfnoc_radio_loopback.cpp—This example uses the RFNoC API to connect an Rx radio to a Tx radio and run a loopback.
:o rfnoc_replay_samples_from_file.cpp—This example uses the replay block to replay data from a file.
:o rfnoc_rx_to_file.cpp— This example uses the RFNoC API to demonstrate the radio receiving and writing to file.
* '''Reference Architecture Example Source Code'''— The following examples, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder, demonstrate synchronized Tx-Rx operation scalable from 2x2 up to 32x32 systems:
:o Arch_iterative_loopback.cpp—This example cycles through each Tx channel to all Rx channels.
:o Arch_multifreq_loopback.cpp—This example cycles through different frequencies and calls rfnoc_txrx_loopback.cpp.
:o Arch_rfnoc_txrx_loopback.cpp—This example demonstrates a single Tx to all Rx loopback file using a multithreaded implementation.
:o Arch_rfnoc_txrx_loopback_mem.cpp—This example demonstrates a single Tx to all Rx loopback to memory using a multithreaded implementation.
:o Arch_rx_to_mem.cpp—This example demonstrates receiving on all channels to memory.
:o Arch_txrx_fullduplex.cpp—This example demonstrates simultaneously transmitting to and receiving from the host.
:o Arch_dynamic_tx.cpp—This example is written for a four-channel system. It demonstrates the ability to stream different files and/or have different logic in each transmit thread. Four (potentially) different transmit threads are spawned, one to each USRP channel. Each thread can be customized as needed. You can apply this example to higher channel count systems, as well as receive threads if needed.

:: '''Note''' The provided examples use the same LO and Tx frequency transmitted across all channels and devices. You must modify this reference architecture if you want to transmit different frequencies across channels.

<span id="building-examples"></span>
== Building Examples ==

The reference architecture examples are compiled when you run the following setup script, setup_script.sh.

If you need to make changes to the examples, you must recompile the code.

# Navigate to ''&lt;REFARCH&gt;/build/'' to recompile the provided examples.
# Execute the following commands in the terminal:

<code>cmake ..

make –j
</code>
If a custom example needs to be compiled, you must modify ''&lt;REFARCH&gt;''/CMakeLists.txt to be used with cmake. To build a new example titled new_example.cpp, complete the following steps.

# Add the following lines to ''&lt;REFARCH&gt;''/CMakeLists.txt:

<code>add_executable(new_example {relative File Location}/new_example.cpp)

message(STATUS &quot;New Example&quot;)

target_link_libraries(new_example PRIVATE UHD_BOOST Arch_lib)
</code>
<ol start="2" style="list-style-type: decimal;">
<li><p>Navigate to ''&lt;REFARCH&gt;/''build/.</p></li>
<li><p>Execute the command <code>make –j </code>in the terminal to compile the custom examples.</p></li></ol>

<span id="configuration-files"></span>
== Configuration Files ==

The reference architecture examples use configuration files instead of input arguments to configure the application. The configuration files are located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/examples/] folder.

* Refer to the example runconfig.cfg file for a detailed explanation of each argument.
* The reference architecture uses a multiple RAID 0 system to increase the number of parallel writes and, thus, achieve higher data rates. You must configure the rx-file-location variable in the runconfig.cfg file. Refer to the configuration file for more information.

:: '''Note''' Ensure that the management address for each USRP is specified in the configuration file as demonstrated in the runconfig.cfg example. Not specifying a management address can lead to loss of performance.

<span id="running-the-examples"></span>
== Running the Examples ==

# Open a terminal in the build folder.
# Run an example using the <code>--cfgFile=runconfig.cfg</code> command. For example, run ''rfnoc_txrx_loopback.cpp'' by executing the following command in a terminal:<br />
<code>./rfnoc_txrx_loopback --cfgFile=../examples/runconfig.cfg</code>

<span id="generating-waveform-data-files"></span>
== Generating Waveform Data Files ==

* The replay block works as a playback buffer that uses DRAM to store samples on the USRP. Refer to the importData function in replaycontrol.cpp in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/lib/''] folder to observe how data is sent from the host to store on the USRP DRAM.
* Sample data files should contain sc16 data samples and should be a multiple of two samples (8 bytes) in size because the replay block records and plays back in multiples of 8 bytes. Refer to samples_generator.py in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;/tools/''] folder for an example of a sine wave generation. Refer to the [https://kb.ettus.com/Using_the_RFNoC_Replay_Block ''Using the RFNoC Replay Block''] Knowledge Base article for more information.

<span id="viewing-data-files"></span>
== Viewing Data Files ==

Received data files consist of interleaved int16 IQ data (IQIQI...). You can view data and helpful phase information with the python-based ''readDatFile.py'' example, located in the [https://github.com/EttusResearch/refarch-multich ''&lt;REFARCH&gt;''/tools/oarer_dat_analysis] folder. Refer to the comments in ''readDatFile.py'' for more information about the use of the program.

<span id="measured-performance"></span>

= Measured Performance =

NI has validated the performance specifications, described in this section, using a 32x32 test system configured as described in [https://kb.ettus.com/Multichannel_RF_Reference_Architecture&action#Building_the_System ''Building the System'']. NI used the reference software to determine the specifications in this document.

<span id="definitions"></span>
== Definitions ==

''Warranted ''specifications describe the performance of the system under stated operating conditions and are covered by the system warranty. Warranted specifications account for measurement uncertainties, temperature drift, and aging. Warranted specifications are ensured by design or verified during production and calibration.

''Characteristics'' describe values that are relevant to the use of the system under stated operating conditions, but are not covered by the model warranty.

''Typical'' specifications describe the performance met by a majority of systems.

''Nominal'' specifications describe an attribute that is based on design, conformance testing, or supplemental testing.

''Measured'' specifications describe the measured performance of a representative system.

Specifications are ''Measured'' unless otherwise noted.

<span id="system-characteristics"></span>
== System Characteristics ==

{| class="wikitable"
|width="49%"| Transmit channel count
|width="50%"| Up to 32
|-
| Receive channel count
| Up to 32
|-
| RF frequency coverage
| 400 MHz
|-
| RF bandwidth
| Up to 200 MHz instantaneous bandwidth per channel
|-
| Configuration plane
| 1 GbE
|-
| Data plane
| 10 GbE
|-
| Processing nodes
|
|-
| On USRP
| FPGA
|-
| On server
| CPU
|-
| Data storage and streaming
|
|-
| Format
| Binary
|-
| Type
| Complex interleaved U16
|-
| Acquisition mode
| Streaming, finite records
|}

<span id="synchronization"></span>
== Synchronization ==

<span id="synchronization-terminology"></span>
=== Synchronization Terminology ===

In the context of this reference architecture, a phase-coherent system is interpreted as a system operating at the same frequency with a consistent phase relationship between any two channels in the system. ''Phase coherence'' is defined as the attribute of two or more waves where the relative phase between waves is constant during the time of the measurement.

Phase coherence performance of the test system is described using the following metrics:

* ''Average channel-to-channel phase skew''—The average phase offset between two channels. Described by how it varies over time or from run to run.
* ''Standard deviation of channel-to-channel skew''—One number that describes the variation of phase offset between channels around the average skew over time or from run to run.
* ''Rx-Rx''—Channel-to-channel specifically referring to Rx channels.

<span id="synchronization-methodology"></span>
=== Synchronization Methodology ===

Skew measurements are made based on IQ data files generated using the reference software provided as part of the Multichannel RF Reference Architecture. The measured system uses the MIMO loopback configuration, described in ''Connecting the RF RX-TX Ports for a MIMO Loopback'', to route signals from all Tx ports to all Rx ports with a consistent path length. To measure the phase skew, the Angle-FFT method is used, wherein the IQ data generated from each channel is measured against the relevant reference signal for each measurement. The FFT method is used to extract the frequency and phase components of each signal in the vicinity of the signal’s fundamental frequency. This information is used to determine the magnitude of the phase offset between two signals in each measurement period. To compare signals throughout their entire duration, each signal is split into shorter measurement windows in which the instantaneous phase is calculated using the method described above.

<span id="rx-phase-coherence-one-hour-stability"></span>
=== Rx Phase Coherence One-Hour Stability<ref>''Stability'' reports variation in the value of average channel-to-channel phase skew over a long (&gt;&gt;1 s) duration.</ref> ===

Measurements are valid under the following conditions:

* Configured carrier frequency: 2 GHz (S-band/L-band)
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 1 hour, sampled at 6.25 MSample/s at receiver (250 MSample/s clock rate with decimation rate of 40)
* Measurement window: 4 ms (2,500 samples)
* Configuration:
* Single device: Stimulus simultaneously received at two Rx channels on the same USRP
* System: Stimulus simultaneously received at two Rx channels on arbitrary separate USRPs
* Analysis:
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range

:: '''Note''' Rx channels on the same USRP internally share the RX LO input to that device as shown in the following figure.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 22. RF Configuration for Stability Measurements
|-
| style="border-style: none;" | [[File:AN822-RF_Configuration_Stability_Diagram.png|431x239px|center]]
|}

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 23. Average Channel-to-Channel Phase Skew between Two Rx Channels on the Same Device over One Hour, Relative to t<sub>0</sub>
|-
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_Device.png|611x224px|center]]
|}

{| class="wikitable"
|width="49%"| Range, single device
|width="50%"| 0.139°
|}

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 24. Average Channel-to-Channel Phase Skew between Two Rx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub>
|-
| style="border-style: none;" | [[File:AN822-Phase_Coherence_Stability Graph_System.png|615x226px|center]]
|}


{| class="wikitable"
|width="49%"| Range, system
|width="50%"| 0.167°
|}

<span id="short-term-rx-rx-phase-coherence"></span>
=== Short-Term Rx-Rx Phase Coherence<ref>''Short-term'' describes variation in channel-to-channel phase skew over a short duration (&lt;100 µs).</ref> ===

Measurements are valid under the following conditions:

* Sample data generation: 32 Rx receiving simultaneously at each carrier frequency
* Carrier frequency range: 1.0 GHz to 5.5 GHz, steps of 100 MHz
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted for 56 µs, sampled at 250 MSample/s
* Measurement window: 8 µs (2,000 samples at 250 MSample/s)
* Analysis:
* Where ''µ<sub>nx</sub>'' is the average skew between channel ''n'' and reference channel in measurement window x
* ''sd<sub>n</sub>'' = σ{''µ<sub>n0</sub>'', ''µ<sub>n1</sub>'',…, ''µ<sub>n6</sub>''} is the standard deviation of average skew for channel n using all measurement windows
* Average over 32 channels is µ{''sd<sub>1</sub>''…}
* Maximum of 32 channels is max{''sd<sub>1</sub>''…}

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 25. Standard Deviation in Short-Term Rx-Rx Phase Skew as a Function of Carrier Frequency
|-
| style="border-style: none;" | [[File:AN822-Short-Term_Phase_Coherence_Graph.png|448x323px|center]]
|}

<span id="repeatability-of-rx-rx-phase-coherence"></span>
=== Repeatability of Rx-Rx Phase Coherence ===

Measurements are valid under the following conditions:

* 10 runs
* Transmit from one Tx to all 32 Rx, repeat 32 times, once with each Tx
* Configured carrier frequency: 2 GHz
* Configured baseband stimulus: 56 µs, sampled at 250 MSample/s
* Measurement window: 8 µs (2,000: samples)
* Analysis:
* Where ''m<sub>nx</sub>'' is Rx n-to-Rx 0 average skew in run ''x''
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Rx-Rx skew for channel ''n''
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} – ''m<sub>n0</sub>'')) is the range from run to run of Rx-Rx skew for channel ''n''
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Rx-Rx skew
* Maximum{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Rx-Rx skew
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Rx-Rx skew
* Maximum{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Rx-Rx skew

<span id="resetting-session-between-runs"></span>
==== Resetting Session between Runs<ref>A ''USRP session'' is a period in which communication links between chips in the USRP are established and not changed. Session reset is accomplished by restarting the reference software for each run.</ref> ====

{| class="wikitable"
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)
!width="50%"|
|-
| Average over 32 channels
| 0.021°
|-
| Maximum of 32 channels
| 0.145°
|-
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)
|
|-
| Average of 32 channels
| 0.070°
|-
| Maximum of 32 channels
| 0.483°
|}

<span id="resetting-configuration-between-runs"></span>
==== Resetting Configuration between Runs<ref>A ''USRP configuration'' is a period in which the digital path within the USRP is configured and not changed. Configuration reset is accomplished by changing carrier frequency from 2.0 GHz to 2.1 GHz and back to 2.0 GHz between runs.</ref> ====

:: '''Note''' In each run analyzed here, the carrier frequency is 2.0 GHz. Users must recharacterize channel-to-channel skew when using different carrier frequencies, because average channel-to-channel skew in the system deviates by &gt;&gt;1° per channel.

{| class="wikitable"
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)
!width="50%"|
|-
| Average over 32 channels
| 0.025°
|-
| Maximum of 32 channels
| 0.066°
|-
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 10 runs (r<sub>n</sub>)
|
|-
| Average of 32 channels
| 0.067°
|-
| Maximum of 32 channels
| 0.169°
|}

<span id="tx-phase-coherence-one-hour-stability"></span>
=== Tx Phase Coherence One-Hour Stability ===

Measurements are valid under the following conditions:

* Configured carrier frequency: 1 GHz
* Configured baseband stimulus: 500 kHz sine wave, continuously transmitted at 250 MSample/s for 1 hour, sampled at 3.2 GSample/s in finite windows
* Measurement window: 35 ms (120,000 samples)
* Configuration:
* Transmitters: Stimulus simultaneously transmitted from any two channels
* Receivers: Stimulus simultaneously received at two Rx channels on external acquisition device
* Analysis:
* Where ''µ<sub>n</sub>'' is average skew between two channels in measurement window ''n''
* ''µ<sub>n</sub>'' – ''µ<sub>0</sub>'' is the average skew relative to ''t<sub>0</sub>''
* Range{''µ<sub>0</sub>'', ''µ<sub>1</sub>'', … , ''µ<sub>N</sub>''} is the range

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 26. Average Channel-to-Channel Phase Skew between Two Tx Channels on the Same Device over One Hour, Relative to t<sub>0</sub>
|-
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_Device.png|431x239px|center]]
|}

{| class="wikitable"
|width="49%"| Range, single device
|width="50%"| 0.128°
|}


{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 27. Average Channel-to-Channel Phase Skew between Two Tx Channels on Separate Devices over One Hour, Relative to t<sub>0</sub>
|-
| style="border-style: none;" | [[File:AN822-Tx_Phase_Coherence_Stability_Graph_System.png|431x239px|center]]
|}

{| class="wikitable"
|width="49%"| Range, system
|width="50%"| 0.115°
|}

<span id="repeatability-of-tx-tx-phase-coherence"></span>
==== Repeatability of Tx-Tx Phase Coherence  ====

Measurements are valid under the following conditions:

* 50 runs
* Transmit from 4 Tx
* Configured carrier frequency: 1 GHz
* Configured baseband stimulus: 35 µs, sampled at 3.4 GSample/s
* Measurement window: 35 µs (120,000: samples)
* Analysis:
* Where ''m<sub>nx</sub>'' is Tx ''n''-to-Tx 0 average skew in run ''x''
* ''sd<sub>n</sub>'' = σ{''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} is the standard deviation from run to run of Tx-Tx skew for channel ''n''
* ''r<sub>n</sub>'' = Max(Abs({''m<sub>n0</sub>'', ''m<sub>n1</sub>'', … ''m<sub>n9</sub>''} - ''m<sub>n0</sub>'')) is the range from run to run of Tx-Tx skew for channel ''n''
* µ{''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system average standard deviation of Tx-Tx skew
* Maximum{ ''sd<sub>1</sub>'', ''sd<sub>2</sub>'',…,''sd<sub>32</sub>''} is the system maximum standard deviation of Tx-Tx skew
* µ{''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system average range of Tx-Tx skew
* Maximum{ ''r<sub>1</sub>'', ''r<sub>2</sub>'',…,''r<sub>32</sub>''} is the system maximum range of Tx-Tx skew

<span id="resetting-session-between-runs-1"></span>
====== Resetting Session between Runs  ======

{| class="wikitable"
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)
!width="50%"|
|-
| Average over 4 channels
| 0.012°
|-
| Maximum of 4 channels
| 0.020°
|-
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)
|
|-
| Average of 4 channels
| 0.039°
|-
| Maximum of 4 channels
| 0.089°
|}

<span id="resetting-configuration-between-runs-1"></span>
====== Resetting Configuration between Runs  ======

{| class="wikitable"
!width="49%"| Run-to-run, standard deviation of channel-to-channel skew (sd<sub>n</sub>)
!width="50%"|
|-
| Average over 4 channels
| 0.014°
|-
| Maximum of 4 channels
| 0.020°
|-
!colspan="2" style="text-align:left;" | Range of average channel-to-channel skew over 50 runs (r<sub>n</sub>)
|
|-
| Average of 4 channels
| 0.041°
|-
| Maximum of 4 channels
| 0.059°
|}

<span id="streaming"></span>

== Streaming ==

<span id="streaming-to-memory-data-transfer-rate"></span>
=== Streaming to Memory Data Transfer Rate<ref>''Streaming to memory'' is measured as the sustained data transfer rate without any overflows.</ref> ===

Measurements are valid under the following conditions:

* Configured frequency: 2.5 GHz (S-band)
* Configured stimulus: Single-tone
* Number of channels: Varied
* Duration: Single run, 10 s



{| class="wikitable" style="text-align: center"
!width="33%"| Streaming Rate (MSample/s per Channel)
!width="33%"| Number of Channels
!width="33%"| Measurement Time (s)
|-
| 33.33
| 32
| 10
|-
| 50
| 16
| 10
|-
| 122.88
| 8
| 10
|}

:: '''Note''' For more information about supported sampling and master clock rates, refer to the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide ''USRP N300/N310/N320/N321 Getting Started Guide''].

<span id="streaming-to-disk-data-transfer-rate"></span>
=== Streaming to Disk Data Transfer Rate<ref>''Streaming to disk'' is measured as the sustained data transfer rate without any overflows.</ref> ===

Measurements are valid under the following conditions:

* Configured frequency: 2.5 GHz (S-band)
* Configured stimulus: Single-tone
* Number of channels: Varied
* Duration: Single run, 10 s



{| class="wikitable" style="text-align: center"
!width="33%"| Streaming Rate (MSample/s per Channel)
!width="33%"| Number of Channels
!width="33%"| Measurement Time (s)
|-
| 33.33
| 32
| 10
|-
| 50
| 16
| 10
|-
| 62.5
| 8
| 10
|}

<span id="streaming-simultaneously-tofrom-disk-rate"></span>
=== Streaming Simultaneously to/from Disk Rate<ref>''Streaming simultaneously to/from disk'' is defined as streaming from the host to the radio for transmission, while simultaneously receiving on the radio to a file on the host. The rates are measured as sustained data transfer rates without any overflows.</ref> ===

Measurements are valid under the following conditions:

* Configured frequency: 2.5 GHz (S-band)
* Configured stimulus: Single-tone
* Number of channels: Varied
* Duration: Single run, 10 s



{| class="wikitable" style="text-align: center"
!width="33%"| Streaming Rate (MSample/s per Channel)
!width="33%"| Number of Channels
!width="33%"| Measurement Time (s)
|-
| 11.11
| 32
| 10
|-
| 25
| 16
| 10
|-
| 62.5
| 8
| 10
|-
| 62.5
| 4
| 10
|}

<span id="appendix-creating-highchannelcount-systems"></span>

= Appendix: Creating High‑Channel‑Count Systems =

NI tested and configured the Multichannel RF Reference Architecture for a 32x32 system. It is possible to build a 64x64 (or greater) system.

<span id="connection-considerations-for-highchannelcount-systems"></span>
== Connection Considerations for High‑Channel‑Count Systems ==

When the system channel count increases to 64 channels and greater, consider the following:

* The MIMO loopback, 10 MHz and PPS, and LO distribution connections require additional tiers.
* Drift increases for typical measurements.

<span id="lo-distribution-for-highchannelcount-systems"></span>
== LO Distribution for High‑Channel‑Count Systems ==

Refer to ''Connecting the LO Distribution'' for information about LO distribution connections and best practices.

:: '''Note''' This configuration is used to synchronize both the Rx and Tx to the same LO resulting in the same center frequencies for both Tx and Rx. To synchronize different frequencies, connect RX LO OUT to RX LO IN and TX LO OUT to TX LO OUT. Refer to [https://kb.ettus.com/USRP_N320/N321_LO_Distribution ''USRP N320/N321 LO Distribution''] for connection diagrams.

The following table lists the tiers for the 64x64 system configuration.



{| class="wikitable" style="text-align: center"
!width="24%"| System
!width="25%"| Tiers
!width="25%"| USRP N321 Minimum Quantity
!width="25%"| USRP N320 Maximum Quantity
|-
| 64x64
| LO Source (0), 1, 2, 3
| 11
| 21
|}

The following diagram represents LO distribution signal connections.

For the 64x64 system, all RX LO OUT cables originating from the LO Source USRP N321 must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 1 USRP N321s must be the same length. All RX LO OUT and TX LO OUT cables originating from the Tier 2 USRP N321s must be the same length. Note that you cannot combine LO signals.

{| class="wikitable" style="text-align: center; border-style: none; background-color:#ffffff;"
|+ Figure 23. LO Sharing (64x64)
|-
| style="border-style: none;" | [[File:AN822-LO_Dist_64X64.png|587x323px|center]]
|}


{| class="wikitable" style="text-align: center"
|+ Table 8. LO Sharing Cabling (64x64)
!width="33%"| Cable Color in Diagram
!width="33%"| Cable Tier
!width="33%"| Number of Cables in Tier
|-
| [[File:AN822-LO_Cable_table_black.png|13x13px]]
| LO Source Tier 0
| 4
|-
| [[File:AN822-LO_Cable_table_blue.png|13x13px]]
| Distribution Tier 1, RX LO OUT
| 8
|-
| [[File:AN822-LO_Cable_table_pink.png|13x13px]]
| Distribution Tier 1, TX LO OUT
| 8
|-
| [[File:AN822-LO_Cable_table_yellow.png|13x13px]]
| Distribution Tier 2, RX LO OUT
| 32
|-
| [[File:AN822-LO_Cable_table_orange.png|13x13px]]
| Distribution Tier 2, TX LO OUT
| 32
|}

<span id="additional-resources"></span>

= Additional Resources =

<span id="related-documentation"></span>
== Related Documentation ==

Refer to the following documents to learn more about the Multichannel RF Reference Architecture, software, and hardware components:

* GitHub repository—Contains the system bill of materials and reference software example and library files needed to set up, install, and run your Multichannel RF Reference Architecture system. Visit https://github.com/EttusResearch/refarch-multich.
* ''USRP N300/N310/N320/N321 Getting Started Guide''—Describes how to install software, how to set up your USRP, and confirm that your device is operating properly. Visit https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide.
* ''N320/N321'' specifications—Lists specifications for the USRP N321 and USRP N320. Visit https://kb.ettus.com/N320/N321.
* ''USRP Hardware Driver and USRP Manual—''Describes how to use the USRP N321 and USRP N320 and how to use the API to connect to devices through your own software. Visit https://files.ettus.com/manual/index.html.
* ''USRP N320/N321 LO Distribution''—Describes LO distribution connections and commands. Visit https://kb.ettus.com/USRP_N320/N321_LO_Distribution.
* ''Using the RFNoC Replay Block''—Describes the basic use of the RFNoC Replay block in UHD and how to run the UHD Replay example. Visit https://kb.ettus.com/Using_the_RFNoC_Replay_Block.

<span id="training-resources"></span>
== Training Resources ==

* Using Open-Source Tools with USRP Hardware for SDR Applications—Visit [https://learn.ni.com/ learn.ni.com], select '''View Library''', and search for <code>using open-source tools with USRP</code>.
* Open-Source SDR Knowledge Base—Visit https://kb.ettus.com/Knowledge_Base.
* SDR-Academy Informal Getting Started Videos—Visit the SDR Academy at [http://www.ettus.com/support/sdr-academy/ www.ettus.com/support/sdr-academy/].
* RFNoC Workshop from GRCon 2020—Visit https://kb.ettus.com/Training_Workshops to find RFNoC video links and workshop materials.

= Footnotes =
<references />