View source for OAI Reference Architecture for 5G and 6G Research with USRP

==Authors==
Neel Pandeya, Bharat Agarwal

==Application Note Number==
'''AN-956'''


==Abstract==

This Application Note presents a reference design for using the Eurecom OpenAirInterface (OAI) software stack to implement 5G NR Stand-Alone (SA) systems on the USRP N300, N310, N320, N321, X410 radio devices. The reference design covers the base station (gNB), the user equipment (UE), and the Core Network (CN) components of the network. Three types of UE are used: a UE running on a USRP radio; a wireless modem module UE; and a commercial (COTS) handset/phone. The reference design supports operation in Frequency Range 1 (FR1), and support for operation in FR2 will be added at a future date. The various aspects of installing, configuring, and running the hardware and software components of the network are discussed in detail, along with a discussion of expected results, methods to benchmark and monitor the system, and troubleshooting steps.

The solution brochure for the OAI Reference Architecture for 5G and 6G Research with the USRP can be downloaded [https://www.ni.com/en/forms/oai-reference-architecture-brochure.html here].

You can learn more about other solutions for 5G and 6G Wireless Research and Prototyping at the [https://www.ni.com/en/solutions/electronics/5g-6g-wireless-research-prototyping.html website here].

== Overview of the USRP N300 and N310 ==

The USRP N300 and N310 are a networked software defined radio that provides reliability and fault-tolerance for deployment in large scale and distributed wireless systems. This device simplifies control and management of a network of radios by introducing the unique capability to remotely perform tasks such as debugging, updating software, rebooting, factory resetting, self-testing, and monitoring system health. The USRP N310 is an all-in-one device that includes two AD9371 transceivers, the Zynq-7100 SoC baseband processor, two SFP+ ports, a built-in GPSDO module, and various other peripheral and synchronization features.

The N300 features:

{|
|style="vertical-align:top"|
* Xilinx Zynq-7035 FPGA SoC
* Dual-core ARM A9 800 MHz CPU
* 2 RX, 2TX in half-wide RU form factor
* 10 MHz – 6 GHz extended frequency range
* Up to 100 MHz of instantaneous bandwidth per channel
* RX, TX filter bank
* 16 bit ADC, 14 bit DAC
* Configurable sample rates: 122.88, 125, and 153.6 MS/s
* Two SFP+ ports (1 GbE, 10 GbE, [[Aurora]])
* RJ45 (1 GbE) 
* 10 MHz clock reference 
* PPS time reference
* Built-in GPSDO 
* 1 Type A USB host port 
* 1 micro-USB port (serial console, JTAG) 
* Watchdog timer 
* OpenEmbedded Linux
* High channel density
* Reliable and fault-tolerant deployment
* Remote management capability
* Stand-alone operation
* USRP N300 '''does not''' contain a Trusted Platform Module 

|[[File:n300_kit.jpg|500px|center]]
|}

The N310 features:

{|
|style="vertical-align:top"|
* Xilinx Zynq-7100 FPGA SoC
* Dual-core ARM A9 800 MHz CPU
* 4 RX, 4TX in half-wide RU form factor
* 10 MHz – 6 GHz extended frequency range
* Up to 100 MHz of instantaneous bandwidth per channel
* RX, TX filter bank
* 16 bit ADC, 14 bit DAC
* Configurable sample rates: 122.88, 125, and 153.6 MS/s
* Two SFP+ ports (1 GbE, 10 GbE, [[Aurora]])
* RJ45 (1 GbE) 
* 10 MHz clock reference 
* PPS time reference
* External RX, TX LO input ports 
* Built-in GPSDO 
* 1 Type A USB host port 
* 1 micro-USB port (serial console, JTAG) 
* Trusted Platform Module (TPM) v1.2
* Watchdog timer 
* OpenEmbedded Linux
* High channel density
* Reliable and fault-tolerant deployment
* Remote management capability
* Stand-alone operation

For more detailed technical information about the USRP N300 and N310, please reference the [https://kb.ettus.com/N320/N321 N300/N310 Hardware Resource Page].

For more information about getting started with the USRP N300 and N310, please reference the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide Getting Started Guide].

|[[File:n310 kit.png|500px|center]]
|}

== Overview of the USRP N320 and N321 ==

The USRP N320 and N321 are a networked software defined radio that provides reliability and fault-tolerance for deployment in large scale and distributed wireless systems. This is a high performance SDR that uses a unique RF design by Ettus Research to provide 2 RX and 2 TX channels in a half-wide RU form factor. Each channel provides up to 200 MHz of instantaneous bandwidth, and covers a frequency range from 3 MHz to 6 GHz. The baseband processor uses the Xilinx Zynq-7100 SoC to deliver a large user programmable FPGA for real-time, low latency processing and a dual-core ARM CPU for stand-alone operation.Support for 1 GbE, 10 GbE, and Aurora interfaces over two SFP+ ports and 1 QSFP+ port enables high throughput IQ streaming to a host PC or FPGA coprocessor. A flexible synchronization architecture with support for LO sharing for TX and RX, 10 MHz clock reference, PPS time reference, GPSDO, and White Rabbit enables implementation of phase coherent MIMO testbeds. The USRP N320 leverages recent software developments in UHD to simplify control and management of multiple devices over the network with the unique capability to remotely administrate tasks such as debugging, updating software, rebooting, resetting to factory state, and monitoring system health.

The USRP N320 features:

{|
|style="vertical-align:top"|
* Xilinx Zynq-7100 FPGA SoC
* Dual-core ARM A9 800 MHz CPU
* 2 RX, 2 TX in half-wide RU form factor
* 3 MHz – 6 GHz frequency range
* Up to 200 MHz of instantaneous bandwidth per channel
* Sub-octave RX, TX filter bank
* 14 bit ADC, 16 bit DAC
* Configurable sample rates: 200, 245.76, 250 MS/s
* Two SFP+ ports (1 GbE, 10 GbE, [[Aurora]], White Rabbit)
* One QSFP+ port ( 2x 10Gb / [[Aurora]] )
* RJ45 (1 GbE) 
* 10 MHz clock reference 
* PPS time reference
* External RX, TX LO input ports 
* Built-in GPSDO 
* 1 Type A USB host port 
* 1 micro-USB port (serial console, JTAG) 
* Trusted Platform Module (TPM) v1.2
* Watchdog timer 
* OpenEmbedded Linux
* Reliable and fault-tolerant deployment
* Remote management capability
* Stand-alone operation

|[[File:n320 kit.png|500px|center]] 

|}

The USRP N321 features:
{|

|style="vertical-align:top"|
* Xilinx Zynq-7100 FPGA SoC
* Dual-core ARM A9 800 MHz CPU
* 2 RX, 2 TX in half-wide RU form factor
* 3 MHz – 6 GHz frequency range
* Up to 200 MHz of instantaneous bandwidth per channel
* Sub-octave RX, TX filter bank
* 14 bit ADC, 16 bit DAC
* Configurable sample rates: 200, 245.76, 250 MS/s
* Two SFP+ ports (1 GbE, 10 GbE, [[Aurora]], White Rabbit)
* One QSFP+ port ( 2x 10Gb / [[Aurora]] )
* RJ45 (1 GbE) 
* 10 MHz clock reference 
* PPS time reference
* External RX, TX LO input ports 
* LO Distribution for up to 128x128 MIMO
* Built-in GPSDO 
* 1 Type A USB host port 
* 1 micro-USB port (serial console, JTAG) 
* Trusted Platform Module (TPM) v1.2
* Watchdog timer 
* OpenEmbedded Linux
* Reliable and fault-tolerant deployment
* Remote management capability
* Stand-alone operation

For more detailed technical information about the USRP N320 and N321, please reference the [https://kb.ettus.com/N320/N321 N320/N321 Hardware Resource Page].

For more information about getting started with the USRP N320 and N321, please reference the [https://kb.ettus.com/USRP_N300/N310/N320/N321_Getting_Started_Guide Getting Started Guide].

|[[File:n321 kit.png|500px|center]] 


|}

== Overview of the USRP X410 ==

The USRP X410 is a high-performance, multi-channel software defined radio. The SDR is designed for frequencies from 1 MHz to 7.2 GHz, tunable up to 8 GHz and features a two-stage superheterodyne architecture with 4 independent TX and RX channels capable of 400 MHz of instantaneous bandwidth each. Digital interfaces for data offload and control include two QSFP28 interfaces capable of 100 GbE[1], a PCIe Gen3 x8 [3] interface, as well standard command, control, and debug interfaces: USB-C JTAG, USB-C console, Ethernet 10/100/1000. The USRP X410 is an all-in-one device built on the Xilinx Zynq Ultrascale+ ZU28DR RF System on Chip (RFSoC) with built-in digital up and down conversion and onboard Soft-Decision Forward Error Correction (SD-FEC) IP.

The USRP X410 features:

{|
|style="vertical-align:top"|
* High channel density
* Reliable and fault-tolerant deployment
* Stand-alone (embedded) or host-based (network streaming) operation
* Fully integrated and assembled (the USRP X410 does not support swappable daughtercards)
* 1 MHz to 7.2 GHz frequency range (tunable up to 8GHz)
* Up to 400 MHz of instantaneous bandwidth per channel
* 4 RX, 4 TX in half-wide RU form factor
* Xilinx Zynq-Ultrascale+ ZU28DR RFSoC
* 12 bit ADC, 14 bit DAC
* IQ Sample Clock rates up to 500 MS/s
* Onboard SD-FEC, DDC, DUC
* Quad-core ARM Cortex-A53 up to 1.2 GHz CPU
* Dual-core ARM Cortex-A5 MPCore up to 500 MHz
* Two QSFP28 ports (10 Gigabit Ethernet, 100 Gigabit Ethernet, Aurora)
* Two iPass+™ zHD® Interfaces (PCIe Gen3 x 8)
* RJ45 (1 GbE) [1]
* 10 MHz Clock reference 
* PPS time reference
* Trig In/Out Interface
* Built-in GPSDO 
* Two FPGA Programmable GPIO Interfaces (HDMI)
* 1 Type C USB host port 
* 1 Type C USB port (serial console, JTAG) 
* Watchdog timer
* OpenEmbedded Linux
* USRP Hardware Driver™ (UHD) open-source software API version 4.1.0 or later
* RF Network on Chip (RFNoC™) FPGA development framework
* Xilinx Vivado® 2019.1 Design Suite (license not included)
* GNU Radio support maintained by Ettus Research™ through GR-UHD, an interface to UHD distributed by GNU Radio
* [1] The RJ45 port is used for remote management of the device and does not support IQ streaming.

For more detailed technical information about the USRP X410, please reference the [https://kb.ettus.com/X410 X410 Hardware Resource Page].

For more information about getting started with the USRP X410, please reference the [https://kb.ettus.com/USRP_X410_Getting_Started_Guide X410 Getting Started Guide].

|[[File:X410.jpg|500px|center]] 


|}

== Overview of the OpenAirInterface (OAI) Software Stack ==

The OpenAirInterface (OAI) software provides an open-source, standards-compliant implementation of a 3GPP 5G NR stack that runs on a commodity x86 CPU and a USRP radio device. OAI was initially developed by Eurecom, which is a university in France.  It is now managed and developed by the OpenAirInterface Software Alliance (OSA), which is a French non-profit organization that provides open-source software and tools for 4G and 5G wireless research.  Further information can be found on the [http://openairinterface.eurecom.fr/ Eurecom website].

The OAI software provides a 5G NR implementation that runs in real-time and is capable of operating with commercial 5G NR handsets (UEs). The OAI software includes implementations of the gNB, the UE, and the Core Network (CN). Further information can be found on the OAI website and their GitLab repository, listed below.

* [https://openairinterface.org/oai-5g-ran-project/ OpenAirInterface 5G Radio Access Network Project]

* [https://gitlab.eurecom.fr/oai/openairinterface5g/ OAI GitLab Repository]

* [https://openairinterface.org/oai-5g-core-network-project/ OAI 5G Core Network]

* [https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-amf/-/wikis/home OAI-CN GitLab repository]

The availability of OAI source code is free for non-commercial and academic research purposes. More information about licensing can be found on the OAI website at [https://openairinterface.org/legal/oai-license-model/ here] and [https://openairinterface.org/legal/oai-public-license/ here].

== Overview of the Reference Architecture ==

The diagram below shows the layout and architecture of this 5G OAI USRP reference architecture.

[[File:Drawing1.png|1000px|center]]

The diagram below shows the RF cable connections between the USRP X410 and the Quectel wireless modem module.

[[File:Drawing0.png|1600px|center]]

The diagram below shows the software layout and architecture across the Core Network, gNodeB, and UE systems.

[[File:Drawing2.png|1200px|center]]

The photos below show one physical implementation of the reference architecture in the office of the authors. Note that this is not the sole valid system implementation, but it is one of many possible valid system implementations.

[[File:Photo_of_system_1.JPG|1200px|center]]

[[File:Photo_of_system_2.JPG|800px|center]]

[[File:Photo_of_system_3.JPG|800px|center]]

[[File:Photo_of_system_4.JPG|800px|center]]

[[File:Photo_of_system_5.JPG|800px|center]]

[[File:Photo_of_system_6.JPG|800px|center]]

[[File:Photo_of_system_7.JPG|800px|center]]

[[File:Photo_of_system_8.JPG|800px|center]]

== Bill of Materials (BoM) ==

The full Bill of Materials (BoM) for the reference design is listed below. The BoM includes all the hardware components for the multiple system configuration scenarios, where the USRP N300, N310, N320, N321, X410 can be used for the gNB and the UE system, and the UE itself can be implemented with a USRP device, or with a wireless modem module, or with a COTS handset.

* Three desktop computers, with Intel Core i9 CPU, of 10th, 11th, or 12th Generation, with clock speed of minimum 4.0 GHz, with minimum 10 physical cores, and also with only NVMe disk drives. See further details about this item in the Hardware Requirements section.

* Two 10 Gbps Ethernet networks cards. We recommend the Intel X710-DA2 and the Nvidia/Mellanox MCX4121A-ACAT ConnectX-4 network cards. See further details about this item in the Hardware Requirements section.

* Two USRP devices.  The USRP may be any of USRP N300, N310, N320, N321, X410. There will be one USRP for the gNB, and one USRP for the UE. The USRP devices can be mixed (i.e., the gNB could run with a USRP X410, while the UE runs with a USRP N310).
** https://kb.ettus.com/N300/N310
** https://kb.ettus.com/N320/N321
** https://kb.ettus.com/X410
** https://www.ettus.com/all-products/usrp-n300/
** https://www.ettus.com/all-products/usrp-n310/
** https://www.ettus.com/all-products/usrp-n320/
** https://www.ettus.com/all-products/usrp-n321/
** https://www.ettus.com/all-products/usrp-x410/

* One QSFP28-to-SFP28 breakout cable (NVIDIA MCP7F00-A003R26N DAC Splitter Cable Ethernet 100GbE to 4x25GbE 3m).  This is only needed when using the USRP X410.
** https://store.mellanox.com/products/nvidia-mcp7f00-a003r26n-passive-copper-splitter-cable-ethernet-100gbe-to-4x25gbe-qsfp28-to-4xsfp28-3m-colored-26awg-ca-n.html
** https://store.mellanox.com/products/nvidia-mcp7f00-a003r30l-passive-copper-splitter-cable-ethernet-100gbe-to-4x25gbe-qsfp28-to-4xsfp28-3m-colored-30awg-ca-l.html

* One OctoClock-G. This is needed to synchronize the gNB USRP and the UE USRP. Ensure that device used is the "-G" model, which contains an internal GPSDO module. This is only needed when the UE is implemented on a USRP device.
** https://kb.ettus.com/OctoClock_CDA-2990
** https://www.ettus.com/all-products/octoclock-g/

* Four 10 Gbps Ethernet cables with SFP+ terminations. These cables are available in 0.5, 1.0, and 3.0 meter lengths. These cables are needed when using the USRP N300, N310, N320, N321, but not when using the USRP X410.
** https://www.ettus.com/all-products/10gige-dc/
** https://www.ettus.com/all-products/10gige-1m/
** https://www.ettus.com/all-products/10gige-3m/

* Four VERT900 antennas and/or four VERT2450 antennas.  You can use either of these antennas based on the bands in use.  There are also many third-party vendors selling a wide variety of antennas, such as omnidirectional antennas.  As long as the antenna has an 50-ohm impedance and has SMA connectors, then it can be used with the USRP radio.
** https://www.ettus.com/all-products/vert900/
** https://www.ettus.com/all-products/vert2450/

* One Quectel RM500Q-GL 5G wireless modem module. See further details about this item in the Hardware Requirements section.
** https://www.quectel.com/product/5g-rm500q-gl
** https://www.quectel.com/product/5g-rm50xq-series

* One Google Pixel 5A 5G handset (phone). Be sure that the handset is unlocked.
** https://www.gsmarena.com/google_pixel_5-10386.php
** https://www.gsmarena.com/google_pixel_5a_5g-11059.php
** https://www.amazon.com/Google-Pixel-5G-Factory-Unlocked/dp/B09DV93S9K/

* Two 5G SIM cards and one USB UICC/SIM card reader/writer.
** https://open-cells.com/index.php/sim-cards/

* One Mini-Circuits 4-way DC-Pass, SMA, Power Splitter, 250 to 6000 MHz, 50Ω (ZN4PD1-63HP-S+).
** https://www.minicircuits.com/WebStore/Splitters.html
** https://www.minicircuits.com/WebStore/dashboard.html?model=ZN4PD1-63HP-S%2B

* Two Mini-Circuits, 2-way, DC-Pass SMA, Power Splitter, 500 to 5000 MHz, 50Ω (ZN2PD2-50-S+).
** https://www.minicircuits.com/WebStore/Splitters.html
** https://www.minicircuits.com/WebStore/dashboard.html?model=ZN2PD2-50-S%2B

* Four Mini-Circuits, VAT-10+, 10 dB Fixed Attenuator, DC to 6000 MHz, 50Ω, SMA.
** https://www.minicircuits.com/WebStore/dashboard.html?model=VAT-10%2B

* Four Mini-Circuits, VAT-20+, 20 dB Fixed Attenuator, DC to 6000 MHz, 50Ω, SMA.
** https://www.minicircuits.com/WebStore/dashboard.html?model=VAT-20%2B

* Four Mini-Circuits, VAT-30+, 30 dB Fixed Attenuator, DC to 6000 MHz, 50Ω, SMA.
** https://www.minicircuits.com/WebStore/dashboard.html?model=VAT-30%2B

* Fourteen Coax SMA Cables, Mini-Circuits, SMA, Hand-Flex Interconnect, 18.0 GHz, 36 inches length (086-36SM+).
** https://www.minicircuits.com/WebStore/dashboard.html?model=086-36SM%2B
** https://www.minicircuits.com/pdfs/086-36SM+.pdf

* One NETGEAR GS108 8-Port Gigabit Ethernet Unmanaged Switch. See further details about this item in the Hardware Requirements section.
** https://www.netgear.com/business/wired/switches/unmanaged/gs108/
** https://www.amazon.com/NETGEAR-Ethernet-Unmanaged-Lifetime-Protection/dp/B00MPVR50A/

* Three adapters from USB 3.0 to 1 Gbps Ethernet. Any major reputable brand will work. Use either USB-A or USB-C connector as per the ports available on your host computer.
** https://www.amazon.com/Network-Adapter-CableCreation-Ethernet-Supporting/dp/B013G4C8RE/
** https://www.amazon.com/Ethernet-Thunderbolt-Gigabit-Network-Compatible/dp/B07XTGKP5M/

== Hardware Requirements ==

=== Host Computers ===

Three host computers are needed, one for the gNB, one for the UE, and one for the CN, with the specifications discussed in this section.  The requirements for the host computer running the CN are not as high as for the gNB and UE, but we recommend that all three host computers meet the requirements described here.  We also strongly recommend that each of the gNB, UE, and CN be implemented on their own dedicated system.  A single host computer should only run the gNB, or the UE, or the CN.

==== CPU ====

We recommend using an Intel Core i9 CPU, or an Intel Xeon CPU, 10th, 11th, or 12th Generation, with minimum clock speed of 4.0 GHz, and with minimum 10 physical cores.  Examples of such a CPU would be the Intel i9-10940X CPU, which has 14 physical cores, 4.60 GHz clock speed, and is 10th generation, and the Intel i9-12900K CPU, which has 16 physical cores, 5.20 GHz clock speed, and is 12th generation, as well as the Intel Xeon Platinum 8351N. Be sure that the CPU has at least 40 PCIe lanes, or at least enough lanes to support any GPU and 10 Gbps Ethernet card that are being used. The system should ideally support PCIe Gen 4.

* https://www.intel.com/content/www/us/en/products/sku/198014/intel-core-i910940x-xseries-processor-19-25m-cache-3-30-ghz/specifications.html

* https://ark.intel.com/content/www/us/en/ark/products/134599/intel-core-i912900k-processor-30m-cache-up-to-5-20-ghz.html

==== Disk ====

We strongly recommend that only NVMe SSD disks are used.  Many systems now use PCIe Gen-4, and we recommend using a PCIe Gen-4 NVMe SSD disk. We recommend the Samsung 980 PRO SSD drive. Do not SATA disks at all. Using a RAID configuration with multiple drives should not be necessary, although this can potentially be an option for further increasing performance and throughput.

* https://www.samsung.com/us/computing/memory-storage/solid-state-drives/980-pro-pcie-4-0-nvme-ssd-1tb-mz-v8p1t0b-am/

* https://www.amazon.com/SAMSUNG-PCIe-Internal-Gaming-MZ-V8P1T0B/dp/B08GLX7TNT/

* https://www.tomshardware.com/reviews/samsung-980-pro-m-2-nvme-ssd-review

==== Memory ====

The system should have either dual-channel or quad-channel DDR4 or DDR5 (preferred) memory, with the highest clock speed available. A minimum of 16 GB or 32 GB of memory should be sufficient. Larger amounts of memory should not be necessary, as no virtualization, RAM disk, or other large in-memory buffering is being used.

==== GPU ====

The GPU does not matter for the purposes of running UHD and OAI.  If you might be doing some AI/ML processing in then GPU, then you may want to use a particular GPU. The OAI 5G stack does not currently leverage the GPU.

==== 10 Gbps Ethernet network card ====

The gNB and UE system will need a two-port 10 Gbps Ethernet network card for connecting to the USRP radio. The CN system does not connect to any USRP, so it does not need a 10 Gbps Ethernet network card.

There are several cards that are relevant and that we recommend, depending on the specific use-case.

The Intel X710-DA2 is a solid network card, and works out-of-the-box with Ubuntu 20.04.4. However, it has some issues with DPDK. This reference design is not initially using DPDK, so these issues are not yet relevant. The card is widely availabe and relatively inexpensive. The X710-DA4 is a four-port version of the card.

Be sure that the computer's chassis can physically accommodate the network card.

* https://ark.intel.com/content/www/us/en/ark/products/83964/intel-ethernet-converged-network-adapter-x710da2.html

* https://www.amazon.com/Intel-Ethernet-Converged-X710-DA2-X710DA2/dp/B00NJ3ZC26/

* https://www.cdw.com/product/intel-ethernet-converged-network-adapter-x710-da2-network-adapter-pcie/3473844

* https://www.newegg.com/intel-x710da2/p/N82E16833106253

The Mellanox MCX4121A-ACAT ConnectX-4 is also a solid network card, and also works out-of-the-box with Ubuntu 20.04.4. Furthermore, it works well with DPDK.

* https://store.mellanox.com/products/nvidia-mcx4121a-acat-connectx-4-lx-en-adapter-card-25gbe-dual-port-sfp28-pcie3-0-x8-rohs-r6.html

* https://www.mellanox.com/files/doc-2020/pb-connectx-4-lx-en-card.pdf

* https://store.nvidia.com/en-us/networking/store/product/MCX4121A-ACAT/nvidiamcx4121a-acatconnectx-4lxenadaptercard25gbe/

* https://www.amazon.com/Mellanox-ConnectX-4-MCX4121A-ACAT-25Gigabit-Ethernet/dp/B011HVAZ78/

* https://www.fs.com/products/119650.html

==== QSFP28-to-SFP28 Breakout Cable for USRP X410 ====

The USRP X410 only has a QSFP28 port, which is for 100 Gbps Ethernet. In order to connect the USRP X410 to the host computer via 10 Gbps Ethernet, a QSFP28-to-SFP28 breakout cable is needed. This cable will be required when using the USRP X410, and will connect directly to the 10 Gbps Ethernet cards.  This cable is not needed for the USRP N300, N310, N320, N321.

It is certainly possible to directly connect the host computer to the 100 Gbps QSFP28 port of the USRP X410.  In order to do this, a QSFP28 100 Gbps Ethernet card would be needed. We recommend the Mellanox/Nvidia MCX516A-CCAT (PCIe Gen3), and the Mellanox/Nvidia MCX516A-CDAT (PCIe Gen4). You will also need a QSFP28 cable. We recommend the Mellanox/Nvidia MCP1600-C003E26N, and the Mellanox/Nvidia MCP1600-C003E30L and MCP7F00-A001R30N. We also recommend the Intel E810 series network cards. There are both 100 Gbps Ethernet QSFP28 cards and 10 Gbps Ethernet SFP28/SFP+ cards. All these cards work well with Ubuntu 20.04 and DPDK.

However, in this release of the reference design, it is not necessary to use 100 Gbps Ethernet, which is only needed for supporting high aggregate data rates, such as for the larger 200 MHz and 400 MHz FR2 channel bandwidths, and/or for 2x2 MIMO configuration, both of which are not yet supported. For this release of the reference design, we recommend using the Intel X710-DA2 and the Mellanox MCX4121A-ACAT ConnectX-4 cards, and using dual 10 Gbps Ethernet connections.

* https://store.nvidia.com/en-us/networking/store/product/MCP7F00-A003R26N/nvidiamcp7f00-a003r26ndacsplittercableethernet100gbeto4x25gbe3m/

* https://store.nvidia.com/en-us/networking/store/product/MCP7F00-A003R30L/nvidiamcp7f00-a003r30ldacsplittercableethernet100gbeto4x25gbe3m/

* https://store.mellanox.com/products/nvidia-mcp7f00-a001r30n-passive-copper-splitter-cable-ethernet-100gbe-to-4x25gbe-qsfp28-to-4xsfp28-1m-colored-30awg-ca-n.html

* https://store.mellanox.com/products/nvidia-mcx516a-ccat-connectx-5-en-adapter-card-100gbe-dual-port-qsfp28-pcie3-0-x16-tall-bracket-rohs-r6.html

* https://store.mellanox.com/products/nvidia-mcx516a-cdat-connectx-5-ex-en-adapter-card-100gbe-dual-port-qsfp28-pcie4-0-x16-tall-bracket-rohs-r6.html

* https://store.mellanox.com/products/nvidia-mcp1600-c003e26n-passive-copper-cable-ethernet-100gbe-qsfp28-3m-black-26awg-ca-n.html

* https://store.mellanox.com/products/nvidia-mcp1600-c003e30l-passive-copper-cable-ethernet-100gbe-qsfp28-3m-black-30awg-ca-l.html

* https://ark.intel.com/content/www/us/en/ark/products/series/184846/100gbe-intel-ethernet-network-adapter-e810.html

* https://www.intel.com/content/www/us/en/products/details/ethernet/800-network-adapters/e810-network-adapters/products.html

* https://www.intel.com/content/www/us/en/products/details/ethernet/800-network-adapters/e810-25gbe-network-adapters/products.html

==== Example Systems ====

There are many vendors who sell host computers that meet these requirements. This reference design was implemented using System 76 Thelio Mira systems and Dell Precision 5820 systems. Both systems have flexible configuration options, and can be configured as described in this section.

* https://system76.com/desktops/thelio-mira-b2/configure

* https://www.dell.com/en-us/work/shop/desktops-all-in-one-pcs/precision-5820-tower-workstation/spd/precision-5820-workstation

=== USRP Devices ===

Two USRP devices are needed, one for the gNB, and one for the UE. The USRP may be any of USRP N300, N310, N320, N321, X410. The USRP devices can be mixed (i.e., the gNB could be implemented with a USRP X410, while the UE could be implemented with a USRP N310). All these USRP devices can support all the channel bandwidths in FR1, up to and including 100 MHz.  For FR2, the USRP N320 can support the 50, 100, 200 MHz channel bandwidths, and the USRP X410 can support all the 50, 100, 200, 400 MHz channel bandwidths.

* https://kb.ettus.com/N300/N310

* https://kb.ettus.com/N320/N321

* https://kb.ettus.com/X410

* https://www.ettus.com/all-products/usrp-n300/

* https://www.ettus.com/all-products/usrp-n310/

* https://www.ettus.com/all-products/usrp-n320/

* https://www.ettus.com/all-products/usrp-n321/

* https://www.ettus.com/all-products/usrp-x410/

=== OctoClock-G ===

One OctoClock-G device is needed to synchronize the gNB USRP and the UE USRP. Ensure that device used is the "-G" version of the OctoClock, which contains an internal GPSDO module. This is only needed when the UE is implemented on a USRP device.

* https://www.ettus.com/all-products/octoclock-g/

* https://kb.ettus.com/OctoClock_CDA-2990

=== Host Computer for 5G Wireless Modem Module ===

One implementation of the UE is a 5G wireless modem module, such as the Quectel RM500Q-GL. The modem module requires a device driver and a connection to a host computer.  It is possible to use one of the gNB, UE, CN host computers to also drive the modem module, but it is recommended to have a separate dedicated fourth host computer for this. This host computer need not be powerful or high-performance, and may run either Ubuntu 20.04 or Windows 10. It may be preferred to use Windows 10, as the Qualcomm drivers may work better on Windows than on Linux.

* https://www.quectel.com/product/5g-rm500q-gl

* https://www.quectel.com/product/5g-rm50xq-series

== Software Requirements ==

=== Operating System ===

The required operating system for the gNB, UE, CN systems is Ubuntu 20.04.4. Be sure to use the Desktop image, not the Server image. It is also necessary to use the low-latency kernel on the gNB and UE systems, but not on the CN system. The kernel version should be 5.15 for Ubuntu 20.04.4. The "Installing and Configuring the UHD Software" section contains detailed information about what specific package dependencies need to be installed, and how to install the low-latency kernel. Either Kubuntu or Xubuntu may also be used, instead of Ubuntu. Do not run Ubuntu in a Virtual Machine (VM). Do not use any virtualization. Be sure to install Ubuntu on-the-metal.

The reference architecture will add support for Ubuntu 22.04.1 in the near future.

* https://releases.ubuntu.com/20.04/
* https://xubuntu.org/download/
* https://kubuntu.org/getkubuntu/
* https://en.wikipedia.org/wiki/Ubuntu
* https://en.wikipedia.org/wiki/Xubuntu
* https://en.wikipedia.org/wiki/Kubuntu

=== UHD ===

UHD is the open-source device driver for all USRP radios. The required version of UHD for this reference design is 4.2.0.0. The "Installing and Configuring the UHD Software" section contains detailed information about the installation and configuration procedure.  The "Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux" Application Note also contains details and thorough information about how to install and configure UHD. We recommend that you build UHD from source code, and do not install it from a binary package. We also recommend that you build UHD first, before building and installing OAI, and that you do not build UHD using the OAI <code> build_oai </code> script.  UHD is required on both the gNB system and the UE system, but it is not needed on the CN system.

* https://github.com/EttusResearch/uhd

=== OpenAirInterface (OAI) ===

The required version of OAI for this reference design is either 2022.w33 or the <code> devel </code> branch. We recommend that you build OAI from source code, using the <code> build_oai </code> script. The OAI software is required on the gNB, UE, and CN systems. The "Installing and Configuring the CN System", "Installing and Configuring the gNB System", and "Installing and Configuring the UE System" sections contains detailed information about the installation and configuration procedure for the OAI software.  Be sure to build and install OAI only after building and installing UHD.

* https://gitlab.eurecom.fr/oai/openairinterface5g

=== DPDK ===

The Data Plane Development Kit (DPDK) is an open-source software project that provides a set of data plane libraries and network interface controller polling-mode drivers for offloading TCP packet processing from the operating system kernel to processes running in user-space. This offloading achieves higher computing efficiency and higher packet throughput than is possible using the interrupt-driven processing provided in the kernel. By putting the network interface driver in user space, avoiding context switches, and pinning I/O threads to cores, UHD and DPDK combine to largely prevent the latency spikes induced by the kernel scheduler, and the overall overhead for packet processing is reduced.

The current version of the reference design does not use DPDK, but it is expected that DPDK will be required in future versions of the reference architecture.

The "Getting Started with DPDK and UHD" Application Note contains detailed information about DPDK.

* https://www.dpdk.org/
* https://github.com/DPDK/dpdk
* https://en.wikipedia.org/wiki/Data_Plane_Development_Kit

== Installing and Configuring the UHD Software ==

This section explains how to build and install UHD from source code. At the time of this writing, we recommend using UHD version 4.3.0.0.

First start by creating a working folder to store Git repositories.

    cd $HOME/git
    mkdir $HOME/git

Next, clone the UHD repository on GitHub in the working folder.

    cd $HOME/git
    git clone http://github.com/EttusResearch/uhd.git

Then, create a build folder, and select UHD version 4.3.0.0.

    cd uhd/host
    mkdir build
    cd build
    git checkout v4.3.0.0

Then, build UHD, using the default settings, and install it to the default location.

    cmake ../
    make -j4
    sudo make install
    sudo ldconfig

Finally, add the following lines the end of your $HOME/.bashrc file.

    export PYTHONPATH=/usr/local/lib/python3/dist-packages:/usr/local/lib/python3.7/site-packages:/usr/local/lib/python3/dist-packages:$PYTHONPATH
    export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

Verify that UHD was correctly installed. Check the version number, and be sure that the uhd_find_devices utility program run.

    uhd_config_info --print-all
    uhd_find_devices
    uhd_usrp_probe

== Installing and Configuring the USRP Radio ==

The USRP N300, N310, N320, N321, X410 can all be used as the gNB and the UE.

=== For the USRP N300 and N310 ===

See the article [[USRP N300/N310/N320/N321 Getting Started Guide]] for information about how to set up and configure your USRP N300 and N310 for use with UHD and OAI.

The USRP N300 and N310 support all the channel bandwidths in FR1.

=== For the USRP N320 and N321 ===

See the article [[USRP N300/N310/N320/N321 Getting Started Guide]] for information about how to set up and configure your USRP N320 and N321 for use with UHD and OAI.

The USRP N320 and N321 support all the channel bandwidths in FR1, and all but the 400 MHz channel bandwidth in FR2.

=== For the USRP X410 ===

See the article [[USRP X410 Getting Started Guide]] for information about how to set up and configure your USRP X410 for use with UHD and OAI.

The USRP X410 supports all the channel bandwidths both in FR1 and FR2.

== Configuring the Ubuntu Linux Operating System  ==

See the article [[USRP Host Performance Tuning Tips and Tricks]] for information about specific settings and configuration procedures needed to enable optimal system performance. Specifically, it is necessary to set the CPU governors, enable the thread priority scheduling, set the socket buffer sizes, set the Ethernet MTU values, and set the network card ring buffer sizes.

The use of the Data Plane Development Kit ([https://www.dpdk.org/ DPDK]) should not be necessary for running any of the FR1 channel bandwidths. At the time of this writing, DPDK is not used in this reference architecture.

== Configuring the BIOS for gNB and UE Systems ==
On the gNB system, and on the UE system, if the UE being used is a USRP radio, there are several specific settings that need to be made. The hyperthreading, CPU frequency control, C-states, P-states, and any other power management should all be disabled. Each BIOS will have a different way to do this. The two screenshots below show where in the BIOS menu to disable C-states and to disable Hyperthreading. Once these options are set in the BIOS, reboot the system.

[[File:c_states.jpg|center]]

[[File:disable-hyperthreading.jpg|center]]

== Installing, Configuring, and Running the CN System ==

This section explains the deployment of the OAI 5G CN system on Ubuntu 20.04 with a generic kernel. The deployment is possible either using docker-compose or Helm Chart. We recommend using docker-compose.

The 5G CN deployment is possible in various forms using docker-compose or Helm Chart.

* Using docker-compose, perform a minimalist deployment
* Using docker-compose, perform a basic deployment
* Using docker-compose, perform a basic-vpp deployment with VPP implementation of UPF
* Using docker-compose, perform a basic deployment with static UE IP address allocation 
* Using Helm Chart, perform a basic deployment
* Using docker-compose, doing network slicing

This Application Note explains only the minimalist deployment using docker-compose. 

The minimalist deployment can be performed in two scenarios:
* Scenario 1: AMF, SMF, UPF (SPGWU), NRF, MYSQL
* Scenario 2: AMF, SMF, UPF (SPGWU), MYSQL

This Application Note only discusses Scenario 1.

The diagram below shows the layout and architecture of the various Docker containers running on the CN machine, along with the IP address of each container.

[[File:cn-docker-containers.jpg|center]]

===Prerequisites for the Scenario 1 Minimalist Deployment===

Verify that you are running docker-compose version 1.25 or higher.

    dpkg --list | grep docker

Verify that you are running Python version 3.6 or higher.

    python3 --version

Note that you should add your username to the <code>docker</code> group so that you can run Docker operations without needing root privileges.

To add your username to the <code>docker</code> group, use the command below.

    sudo usermod -a -G docker yourusername


In order to pull the Docker images for the different network functions of the 5G CN, you need to have an account on docker-hub. Use this [https://hub.docker.com/ link] to create an account.

===Pull Base Images===

We need to pull two base Docker images <code>ubuntu:bionic</code> and <code>mysql:5.7</code>.

Log in with your Docker Hub credentials, and push and pull images from Docker Hub.

    docker login
    docker pull ubuntu:bionic
    docker pull mysql:5.7
    docker logout

Be sure to set the following network configuration settings.  You must run these commands every time you restart the CN machine, as they will not persist across a reboot.

    sudo sysctl net.ipv4.conf.all.forwarding=1
    sudo iptables -P FORWARD ACCEPT

===Pulling the images from Docker Hub===

The images are hosted under the OAI account <code>oaisoftwarealliance</code>.

You may need to login to Docker Hub.

    docker login

Pull the images listed below.

    docker pull rdefosseoai/oai-amf:latest
    docker pull rdefosseoai/oai-nrf:latest
    docker pull rdefosseoai/oai-spgwu-tiny:latest
    docker pull rdefosseoai/oai-smf:latest
    docker pull rdefosseoai/oai-udr:latest
    docker pull rdefosseoai/oai-udm:latest
    docker pull rdefosseoai/oai-ausf:latest
    docker pull rdefosseoai/oai-upf-vpp:latest
    docker pull rdefosseoai/oai-nssf:latest

Re-tag the images so that the docker-compose files for the tutorial work.

    docker image tag rdefosseoai/oai-amf:latest oai-amf:latest
    docker image tag rdefosseoai/oai-nrf:latest oai-nrf:latest
    docker image tag rdefosseoai/oai-smf:latest oai-smf:latest
    docker image tag rdefosseoai/oai-spgwu-tiny:latest oai-spgwu-tiny:latest
    docker image tag rdefosseoai/oai-udr:latest oai-udr:latest
    docker image tag rdefosseoai/oai-udm:latest oai-udm:latest
    docker image tag rdefosseoai/oai-ausf:latest oai-ausf:latest
    docker image tag rdefosseoai/oai-upf-vpp:latest oai-upf-vpp:latest
    docker image tag rdefosseoai/oai-nssf:latest oai-nssf:latest

Finally, you may logoff.  Your token is stored in plain-text.

   docker logout

===Synchronizing the installed images===

Clone the Git repository for the Core Network, and checkout the <code>v1.3.0</code> tag.

    git clone --branch v1.3.0 https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-fed.git

You may also do this in two discrete steps.

    git clone https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-fed.git
    git checkout v1.3.0

Next, go into the top-level folder in the repository.

    cd oai-cn5g-fed


Next, synchronize all the Git submodules.

    ./scripts/syncComponents.sh

    ---------------------------------------------------------
    OAI-NRF     component branch : master
    OAI-AMF     component branch : master
    OAI-SMF     component branch : master
    OAI-SPGW-U  component branch : master
    OAI-AUSF    component branch : master
    OAI-UDM     component branch : master
    OAI-UDR     component branch : master
    OAI-UPF-VPP component branch : master
    OAI-NSSF    component branch : master
    ---------------------------------------------------------

    git submodule deinit --all --force
    git submodule init
    git submodule update

===Configure the Containers===

Configure the Core Network by editing the <code>docker-compose-mini-nrf.yaml</code> file and specifying the proper PLMN, TAC, Operator Key, and DNN, according to the gNB and the UE being used.

If you are using the <code>v1.3.0</code> tag for the CN code, then specify the PLMN, TAC, and Operator Key in the <code>amf.conf</code> file, according to the gNB and the UE being used.  If you are using the newer <code>v1.3.0</code> tag for the CN code, then do not do this.

User subscription information should be present in the MySQL database before trying to connect the UE. This can be done by adding the UE information into the <code>oai_db1.sql</code> file. Adjust the values shown below with your actual values.

    INSERT INTO users VALUES
        (imsi,msisdn,imei,NULL,'PURGED',50,40000000,100000000,47,0000000000,1,key,0,0,0x40,'ebd07771ace8677a',opc);

===Deploy the containers===

The containers must be deployed in a strict order for AMF, SMF, and UPF registration with NRF.  The data flow sequence must be from the MySQL database, to OAI-NRF, to OAI-AMF, to OAI-SMF, to OAI-UPF.

===Installing docker-compose and Wireshark=== 

Install <code>docker-compose</code> onto the Core Network system. The link below provides detailed instructions.

    https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-compose-on-ubuntu-20-04

Install Wireshark with the commands listed below.

    sudo add-apt-repository ppa:wireshark-dev/stable
    sudo apt update
    sudo apt install wireshark

Check the Wireshark version, and confirm that it is at least version 3.4.7.

    wireshark --version

=== Invoking the Core Network===

The Core Network is invoked using a Python script, which is a wrapper for docker-compose and other Docker commands. The script informs the user when the Core Network is correctly configured by checking the health status of containers and checking the connectivity between different Core Network components. Run the commands below.

    cd oai-cn5g-fed/docker-compose
    sudo  python3 ./core-network.py --type start-mini --fqdn no --scenario 1

After running the command, you should see the output listed below.

    root:DEBUG:  Starting 5gcn components... Please wait....
    Creating oai-nrf ... done
    Creating mysql   ... done
    Creating oai-amf ... done
    Creating oai-smf ... done
    Creating oai-spgwu ... done
    Creating oai-ext-dn ... done
    [2021-09-14 16:44:10,098] root:DEBUG:  OAI 5G Core network started, checking the health status of the containers... takes few secs....
    [2021-09-14 16:44:47,025] root:DEBUG:  All components are healthy, please see below for more details....
    Name                 Command                  State                  Ports            
    -----------------------------------------------------------------------------------------
    mysql        docker-entrypoint.sh mysqld      Up (healthy)   3306/tcp, 33060/tcp         
    oai-amf      /bin/bash /openair-amf/bin ...   Up (healthy)   38412/sctp, 80/tcp, 9090/tcp
    oai-ext-dn   /bin/bash -c  apt update;  ...   Up                                         
    oai-nrf      /bin/bash /openair-nrf/bin ...   Up (healthy)   80/tcp, 9090/tcp            
    oai-smf      /bin/bash /openair-smf/bin ...   Up (healthy)   80/tcp, 8805/udp, 9090/tcp  
    oai-spgwu    /openair-spgwu-tiny/bin/en ...   Up (healthy)   2152/udp, 8805/udp
    [2021-09-14 16:44:47,025] root:DEBUG:  Checking if the containers are configured....
    [2021-09-14 16:44:47,025] root:DEBUG:  Checking if SMF and UPF registered with nrf core network....
    [2021-09-14 16:44:47,059] root:DEBUG:  For example: oai-smf Registration with oai-nrf can be checked on this url /nnrf-nfm/v1/nf-instances?nf-type="SMF" {"_links":{"item":[{"href":"192.168.70.133"}],"self":""}}....
    [2021-09-14 16:44:47,059] root:DEBUG:  SMF and UPF are registered to NRF....
    [2021-09-14 16:44:47,059] root:DEBUG:  Checking if SMF is able to connect with UPF....
    [2021-09-14 16:44:47,176] root:DEBUG:  UPF receiving heathbeats from SMF....
    [2021-09-14 16:44:47,176] root:DEBUG:  OAI 5G Core network is configured and healthy....

After launching the OAI 5G Core Network, run Wireshark from another terminal.

    sudo wireshark

Once Wireshark has launched, you should see the <code>demo-oai</code> network interface listed. Open the <code>demo-oai</code> network interface, and confirm that all the packets exchanged between the different containers such as NRF, AMF, SMF, UPF are shown.

[[File:screenshot-wireshark-interface-demo-oai.png|center]]

The <code>demo-oai</code> network interface can also be created manually. Since this is not the default behavior, you would need to edit the <code>docker-compose</code> file. The bottom section of the <code>docker-compose-mininrf.yaml</code> should be configured as shown below.

    networks:
        public_net:
            external:
                name: demo-oai-public-net
        # public_net:
        #     driver: bridge
        #     name: demo-oai-public-net
        #     ipam:
        #         config:
        #             - subnet: 192.168.70.128/26
        #     driver_opts:
        #         com.docker.network.bridge.name: "demo-oai"

The <code>docker-compose-host</code> machine needs to be configured with the <code>demo-oai</code> network interface before deploying the core network components. This is needed in order to capture the initial message exchange between the SMF-to-NRF-to-UPF.

    (docker-compose-host)v$ docker network create \
      --driver=bridge \
      --subnet=192.168.70.128/26 \
      -o "com.docker.network.bridge.name"="demo-oai" \
      demo-oai-public-net
    455631b3749ccd6f10a366cd1c49d5a66cf976d176884252d5d88a1e54049bc5
    (docker-compose-host)$ ifconfig demo-oai
    demo-oai: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
            inet 192.168.70.129  netmask 255.255.255.192  broadcast 192.168.70.191
            RX packets 0  bytes 0 (0.0 B)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 0  bytes 0 (0.0 B)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    (docker-compose-host)$ docker network ls
    NETWORK ID          NAME                  DRIVER              SCOPE
    d2d34e05bb2d        bridge                bridge              local
    455631b3749c        demo-oai-public-net   bridge              local

We will next create a bridge automatically. The bridge can be automatically created using the <code>docker-compose</code> file, if there is no need to capture initial packets. This is the default version in the <code>docker-compose-mini-nrf.yaml</code> file. The bottom section should be configured as shown below.

    networks:
        # public_net:
        #     external:
        #         name: demo-oai-public-net
          public_net:
              driver: bridge
              name: demo-oai-public-net
              ipam:
                  config:
                      - subnet: 192.168.70.128/26
              driver_opts:
                  com.docker.network.bridge.name: "demo-oai"

The screenshots listed below show the Wireshark packet captures for various components when the bridge is built manually.

Showing the NRF and AMF:
[[File:screenshot-wireshark-capture-NRF-and-AMF.png]]

Showing the NRF and SMF:
[[File:screenshot-wireshark-capture-NRF-and-SMF.png]]

Showing the SMF and UPF:
[[File:screenshot-wireshark-capture-SMF-and-UPF.png]]

Showing the NRF and UPF:
[[File:screenshot-wireshark-capture-NRF-and-UPF.png]]

Showing the SMF request to NRF for subscribing UPF registration/deregistration events:
[[File:screenshot-wireshark-capture-SMF-request-NRF.png]]

Showing the SMF registration with NRF for the PUT request:
[[File:screenshot-wireshark-capture-SMF-registration-NRF.png]]

Showing the UPF registration with NRF for the PUT request:
[[File:screenshot-wireshark-capture-UPF-registration-with-NRF.png]]

Showing the NRF notification to SMF for UPF registration for the POST request:
[[File:screenshot-wireshark-capture-NRF-notification.png]]

Showing the SMF-to-UPF PFCP association setup request-and-response:
[[File:screenshot-wireshark-capture-SMF-UPF-PFCP.png]]

These Wireshark packet captures are only visible if you create the <code>demo-oai</code> bridge manually.  If the bridge is built automatically, then you will not observe these Wireshark packet captures.

===Stopping the Core Network===

In order to halt the operation of the Core Network, run the commands listed below.

    cd oai-cn5g-fed/docker-compose 
    sudo python3 ./core-network.py --type stop-mini --fqdn no --scenario 1

== Installing, Configuring, and Running the gNB System ==

Before installing and configuring the gNB system, be sure that you first have configured the system BIOS, implemented the Linux performance tuning settings, and installed UHD.

===Building and installing the gNB software===

The commands listed below build and install the gNB software from source code using the OAI Git repository.

    git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git
    cd openairinterface5g
    git checkout 2022.w33
    source oaienv
    cd cmake_targets

Confirm that you are using the correct branch and commit hash. Run the command listed below, and verify that the branch is <code>2022.w33</code> and that the commit hash is <code>ad8381a66bb67d6ef7fa94c9a6ae6c66f3ae84b8</code>.

    git status

Edit the <code>build_oai</code> script, and comment out the lines shown below, by adding a hash mark <code>#</code> at the beginning of each line.

    if [ "$HW" == "OAI_USRP" ] ; then
      echo_info "installing packages for USRP support"
      #check_install_usrp_uhd_driver
      #if [ ! "$DISABLE_HARDWARE_DEPENDENCY" == "True" ]; then
      #  install_usrp_uhd_driver $UHD_IMAGES_DIR
      #fi
    fi

For the very first time that you build the gNB software, use the command listed below, which includes the "-I" option, which installs the package dependencies for the gNB software.

    ./build_oai -I --w USRP

For successive builds, use the command listed below, which omits the "-I" option.

    ./build_oai --gNB --w USRP

===Editing the gNB configuration file===

Once the gNB software is built and installed, there are specific settings that need to be made in the gNB configuration file before we can run the gNB software. The folder <code> openairinterface5g/ci_scripts/conf_files </code> contains various configuration files. For this Application Note, we will use the configuration file <code> band78.sa.fr1.106PRB.usrpn310.conf </code>. In this file, change the parameters in the PLMN section. Ensure that the PLMN configuration values are the same as the values used for the CN system, specifically <code> tracking_area_code </code>, <code> mcc </code>, <code> mnc </code>, <code> sst </code>, and <code> sd </code>.

[[File:figure-gnb-config-file-mcc-mnc_ver2.jpg|800px|center]]

===Specifying the AMF IP address===

The correct IP address for the AMF needs to be defined in the configuration file, as shown below. In the configuration file, in the <code>AMF parameters</code> section, in the <code>ipv4</code> field, add the IP address of the AMF Docker container. In this reference architecture, it is 192.168.70.132.

[[File:figure-gnb-config-file-amf-parameters_ver2.jpg|600px|center]]

===Specifying the gNB IP address===

The correct IP address for the gNB needs to be defined in the configuration file, as shown below. In the configuration file, in the <code>NETWORK_INTERFACES</code> section, add the IP address of the gNB system to the <code>GNB_IPV4_ADDRESS_FOR_NG_AMF</code> and the <code>GNB_IPV4_ADDRESS_FOR_NGU</code> fields. In this reference architecture, it is 10.89.14.37.

[[File:figure-config-file-gnb-ip-addresses.png|center]]

===Specifying the USRP IP address===

The correct IP address for the USRP needs to be defined in the configuration file, as shown below. In the configuration file, in the <code>RU</code> section, in the <code>sdr_addrs</code> field, we need to specify the device arguments for the USRP, which includes the type of the USRP (<code>type</code>), the mangement IP address (<code>mgmt_addr</code>), the primary streaming IP address (<code>addr</code>), the secondary streaming IP address (<code>second_addr</code>), the clocking source (<code>clock_source</code>), and the timing source (<code>time_source</code>). In this reference architecture, we use the device argument string listed below.

    sdr_addrs="type=x4xx,mgmt_addr=192.168.10.2,addr=192.168.10.2,second_addr=192.168.20.2,clock_source=external,time_source=external"

The <code>mgmt_addr</code>, <code>addr</code>, and <code>second_addr</code> should be defined according to the configuration of your USRP device. For the <code>clock_source</code> and the <code>time source</code>, the values can be <code>internal</code>, <code>external</code>, or <code>gpsdo</code>.  In this reference architecture, an OctoClock-G is connected to the USRP, so we use the value of <code>external</code> for both the clocking source and the timing source.

===Adding a static route to the CN system===

To enable communication between the gNB and CN machine, a static route must be added to the gNB machine. Note that this added route is not permanent, and when the system is restarted, the static route will need to be added again.  The command listed below will add the static route.

    sudo ip route add 192.168.70.128/26 via 10.89.14.119 dev eno1

In this command, the IP address 192.168.70.128/26 is for the <code>demo-oai</code> network bridge created when the CN is invoked, and the IP address 10.89.14.119 is for the CN machine, and the <code>eno1</code> Ethernet interface is the port on the gNB machine connected to the CN machine.

On your machine, the Ethernet interface name will likely be different, but you can use the same IP address for the CN machine, and the same static IP address for the CN machine, as used in this reference architecture.

===Invoking the gNB===

In order to invoke the gNB, we first need to copy the edited gNB configuration file from the folder <code>openairinterface5g/ci_scripts/conf_files</code> to the folder <code>openairinterface5g/targets/PROJECTS/GENERIC-NR-5G/CONF</code>.  Then, the command listed below will invoke the gNB.

    sudo ./nr-softmodem -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb.band78.sa.fr1.106PRB.usrpn310.conf --sa --usrp-tx-thread-config 1 --thread-pool 0,2,4,6

Launch Wireshark, select the Ethernet interface connected to the CN machine, and watch for NGAP packets. As soon as the gNB is launched, the CN and gNB will exchange NGAP setup request and response messages. In these messages, the gNB and CN check the MCC, MNC and TAC parameters. If these values are identical on both the gNB machine and the CN machine, then the NGAP setup request and response will be successful.

[[File:GNB_Fig_9_NGAP_packets_ver2.jpg|800px|center]]

===Interfaces Between gNB and Core Network===

When the gNB is invoked, the N2 and N3 interfaces will be created. The N2 interface supports the NGAP protocol, and the N3 interface supports the GTP protocol. The N1 interface is created when UE attaches to the network and supports the NAS protocol.

[[File:GNB_Fig_7_gNB_interface_with_core_network_ver2.jpg|800px|center]]

===Testing Connectivity Between gNB and CN===

Once both the gNB and the CN machines are up-and-running, test the connectivity between the two systems by pinging the AMF in the CN machine from the gNB machine. On the gNB machine, run the command listed below.

    ping 192.168.70.132 

If the ping fails, then it may be due to the firewall settings.  On the CN machine, run the commands listed below to configure the firewall.

    sysctl net.ipv4.conf.all.forwarding=1
    sudo iptables -P FORWARD ACCEPT 

After running these commands on the CN machine, run this command on the gNB machine to add a route from the gNB machine to the CN machine.  You will likely need to replace the Ethernet interface name <code>eno1</code> with the correct interface name on your gNB machine.

    sudo ip route add 192.168.70.128/26 via 10.89.14.6 dev eno1 

The IP address 192.168.70.128/26 is for the <code>demo-oai</code> interface, and 10.89.14.6 is the IP address of the CN machine, and <code>eno1</code> is the Ethernet interface name on the gNB machine.

[[File:GNB_Fig_8_Verification_Steps_ver2.jpg|800px|center]]

[[File:GNB_Fig_10_SCTP_packets_ver2.jpg|800px|center]]

== Installing, Configuring, and Running the UE System ==

There are three scenarios for the UE implementation: the USRP radio; the 5G wireless modem module; and the COTS handset.

=== UE Scenario 1: USRP ===

The UE can be implemented by running the OAI softmodem with a USRP N300, N310, N320, N321, or X410 device. Currently, this reference architecture is focused on the use of a USRP X410 device, although the integration of the other USRP devices is similar to the integration of the USRP X410 and is relatively straight-forward.

Before installing and configuring the UE system, be sure that you first have configured the system BIOS, implemented the Linux performance tuning settings, and installed UHD.

====Building and installing the OAI softmodem software====

Note that the procedure for building and configuring the UE software is very similar to the procedure for building and installing the gNB software.

The commands listed below build and install the UE software from source code using the OAI Git repository.

    git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git
    cd openairinterface5g
    git checkout 2022.w33
    source oaienv
    cd cmake_targets

Confirm that you are using the correct branch and commit hash. Run the command listed below, and verify that the branch is <code>2022.w33</code> and that the commit hash is <code>ad8381a66bb67d6ef7fa94c9a6ae6c66f3ae84b8</code>.

    git status

Edit the <code>build_oai</code> script, and comment out the lines shown below, by adding a hash mark <code>#</code> at the beginning of each line.

    if [ "$HW" == "OAI_USRP" ] ; then
      echo_info "installing packages for USRP support"
      #check_install_usrp_uhd_driver
      #if [ ! "$DISABLE_HARDWARE_DEPENDENCY" == "True" ]; then
      #  install_usrp_uhd_driver $UHD_IMAGES_DIR
      #fi
    fi

For the very first time that you build the gNB software, use the command listed below, which includes the "-I" option, which installs the package dependencies for the gNB software.

    ./build_oai -I --w USRP

For successive builds, use the command listed below, which omits the "-I" option.

    ./build_oai --gNB --w USRP

====Editing the UE Configuration File====

Once we build and install the UE software, we need to modify the UE configuration file. Only one UE configuration file is available in the folder <code>openairinterface5g/targets/PROJECTS/GENERIC-NR-5GC/CONF</code>. The UE configuration file used in this reference architecture is <code>ue.conf</code>. The default contents of the file is shown below.

    uicc0 = {
    imsi = "2089900007487";
    key = "fec86ba6eb707ed08905757b1bb44b8f";
    opc= "C42449363BBAD02B66D16BC975D77CC1";
    dnn= "oai";
    nssai_sst=1;
    nssai_sd=1;
    }

Edit the fields <code>imsi</code>, <code>key</code>, <code>opc</code>, <code>dnn</code>, <code>nssai_sst</code>, <code>nssai_sd</code> per the values for the CN. Earlier, when configuring the Docker containers for the CN system, we entered several parameters in the <code>oai_db1.sql</code> file. The <code>imsi</code>, <code>key</code>, <code>opc</code> fields there should match the values here in the UE configuration file. The <code>dnn</code> value should be same as listed in the <code>docker-compose-mini-nrf.yaml<code> file. The <code>nssai_sst</code> and <code>nssai_sd</code> values should be same as listed in the gNB configuration file and the <code>docker-compose-mini-nrf.yaml</code> file.

====Invoking the OAI UE softmodem====

Run the command listed below to invoke the OAI UE softmodem.

    openairinterface5g/cmake_targets/ran_build/build sudo ./nr-uesoftmodem --usrp-args "addr=192.168.10.2, clock_source=external,time_source=external" -r 106 --numerology 1 --band 78 -C 3319680000 --sa --nokrnmod 1 -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/ue.conf

The command-line option <code>--usrp-args</code> specifies the USRP device arguments, which are passed to the UHD driver. Here we specify the IP address of the USRP, along with clocking source and the timing source, which are set to <code>external</code>, to indicate the use of the connection to the OctoClock-G device.

The command-line option <code>-r 106</code> specifies the PRBS being used. Based on the PRBS, we can calculate the operational channel bandwidth being used (i.e., 40 MHz).

The command-line option <code>--numerology 1</code> specifies the SCS being used (i.e., 30 KHz).

The command-line option <code>--band 78</code> specifies the 3GPP frequency band being used.

The command-line option <code>-C 3319680000</code> specifies the CF we are operating (i.e., 3.319 GHz).

The command-line option <code>--nokrnmod 1</code> forces the use of the tunnel interface.

Once the UE is attached to the CN, verify that the <code>oaitun_ue1</code> network interface has been created, and that is has been assigned an IP address of 10.x.x.x, using the command listed below.

    ifconfig

The call attach process between the UE and the gNB is illustrated in the sequence diagram shown below.

[[File:oai-ue-call-attach-process-diagram.jpg|700px|center]]

The relevant lines in the OAI UE softmodem log file show information about the call attach process.

[[File:oai-ue-logs.jpg|800px|center]]

Launch Wireshark to view the packets exchanged between the UE and gNB for this call attach process.

[[File:oai-ue-wireshark-captures-between-ue-and-gnb.jpg|800px|center]]

====Verify that the UE was connected to the gNB====

To verify that UE has successfully attached to the gNB, check the OAI-AMF logs on the CN machine using the command listed below. Check the output to confirm that UE has been registered by the Core Network.

    sudo docker logs oai-amf

Run <code>ifconfig</code> on the UE machine to check the IP address assigned to the UE. Verify that the <code>oaitun_ue1</code> network interface has been created, and that is has been assigned an IP address of 10.x.x.x. You can also check the OAI-SMF logs using the command listed below.

    sudo docker oai-smf logs

=== UE Scenario 2: 5G Wireless Modem Module ===

The UE can be implemented using a 5G wireless modem module. Currently, this reference architecture is focused on the use of a Quectel RM500Q-GL 5G wireless modem module. This device is a 5G NR sub-6GHz M.2 module which meets the 3GPP Release 15 specification, and is optimized for industrial and commercial IoT and eMBB applications.  It supports both standalone (SA) and non-standalone (NSA) modes. Other wireless modem modules can certainly be used as well.  Detailed documentation for the [https://www.sierrawireless.com/iot-modules/5g-modules/em9191/ Sierra Wireless EM9191] module will be added in the near future.

====Configuring the SIM Card====

When using a 5G wireless modem module or a COTS handset, a SIM card will be required. If a USRP is being used as the UE, running the OAI UE softmodem, then a SIM card is not required.

The SIM card used in this reference architecture is provided by [https://open-cells.com/index.php/sim-cards/ Open-Cells], and is shown below. Note that the ADM code is printed directly on the SIM card itself.

[[File:quectel-ue-sim-card.jpg|500px|center]]

[[File:quectel-ue-ADM-code.jpg|600px|center]]

Insert the nano SIM card into the SIM card reder/writer, and plug it into the USB slot on the UE computer.

To read and program the SIM card, we use the program <code>program_uicc</code> from Open-Cells ([https://open-cells.com/index.php/uiccsim-programing/ here]).

We first read the existing data on the SIM by running the command below.

    sudo ./program_uicc --adm 1

[[File:quectel-ue-program_uicc_output_for_read.jpg|800px|center]]

We then write the key and the OPC in the UICC file in the SIM card.  The ADM value enables this. Run the command below to perform this operation, where <code>ADM_VALUE_FROM_SIM</code> is the ADM value printed directly on the SIM card itself.

    sudo ./program_uicc --adm <ADM_VALUE_FROM_SIM> --key 0C0A34601D4F07677303652C0462535B --opc 63bfa50ee6523365ff14c1f45f88737d --authenticate --noreadafter 

[[File:quectel-ue-program_uicc_output_for_write.jpg|800px|center]]

Ensure that the values being programmed into the SIM card match the corresponding values entered in the SQL database on the CN machine. The values of primary importance are listed in the table below.

{| class="wikitable" style="margin:auto; text-align: center;"
|+ Primary Configuration Parameters for UE, gNB, CN
|-
! Parameter !! UE !! gNB !! CN
|-
| style="width: 10%"| <b>IMSI</b> || style="width: 30%"| <code>208920100001101</code> || style="width: 30%"| MCC: <code>208</code>, MNC: <code>92</code> || style="width: 30%"| <code>208920100001101</code>
|-
| style="width: 10%"| <b>MSISDN</b> || style="width: 30%"| <code>00000101</code> ||  || style="width: 30%"| <code>00000101</code>
|-
| style="width: 10%"| <b>IMEI</b> || style="width: 30%"| <code>863305040549338</code> ||  || style="width: 30%"| <code>863305040549338</code>
|-
| style="width: 10%"| <b>Key</b> || style="width: 30%"| <code>0C0A34601D4F07677303652C0462535B</code> ||  || style="width: 30%"| <code>0C0A34601D4F07677303652C0462535B</code>
|-
| style="width: 10%"| <b>OPC</b> || style="width: 30%"| <code>63bfa50ee6523365ff14c1f45f88737d</code> ||  || style="width: 30%"| <code>63bfa50ee6523365ff14c1f45f88737d</code>
|-
|}

====Serial Connection to the Module via Minicom====

The The The

Attach all four antennas to the Quectel wireless modem module. Then, mount the Quectel module into the M.2 connector slot on the carrier board.  Then, connect the carrier board to the UE computer via a USB 3.0 port.

We will use Minicom to communicate with the Quectel module over a USB serial connection. Run <code>which minicom</code> to verify that Minicom is already installed.  If not, then run the command listed below to install it.

    sudo apt-get install minicom

Once the Quectel module is plugged in, the Linux operating system should create several USB serial devices which can be used to communicate with the module.  The default device should be <code>/dev/ttyUSB0</code>. Run the command listed below to start a Minicom serial console session with the Quectel device.

    sudo minicom /dev/ttyUSB0

Note that in order to exit Minicom, type Ctrl-A, then "X".

====AT Commands for the Module====

We use Minicom to issue AT Commands to the 5G modem module.

There are informative articles about AT Commands [https://en.wikipedia.org/wiki/Hayes_command_set here] and [https://www.twilio.com/docs/iot/supersim/introduction-to-modem-at-commands here].

There are some specific AT Commands that we need to control and query the Quectel module. The AT Command are generally not case-sensitive. Execute all the AT Commands listed below, in-order, to setup and configure the Quectel module.

; <code>AT+GMR</code>
: Display the current firmwave version number.

; <code>AT+CIMI</code>
: Display the IMSI of the (U)SIM.

; <code>AT+GSN</code>
: Display the IMEI of the (U)SIM.

; <code>AT+QMBNCFG="select","ROW_Commercial"</code>
: Unlock the Quectel module.

; <code>AT+qnwprefcfg="nr5g_band"</code>
: Display which 5G NR frequency bands are configured.

; <code>AT+qnwprefcfg="mode_pref"</code>
: Display which 5G NR mode is set.

; <code>AT+qnwprefcfg="mode_pref",nr5g</code>
: Set the operational mode to 5G NR SA.

; <code>AT+qnwprefcfg="nr5g_disable_mode",0</code>
: Enable 5G NR operations.

; <code>AT+cgdcont=1,“IP“,“oai“,“0.0.0.0“,0,0 (cid, pdp type, apn, pdp_addr, data_comp, head-comp)</code>
: Specify the PDP context parameters for a specific context cid.

; <code>AT+CFUN=0</code>
: Set the the minimum functionality of the module.

; <code>AT+CFUN=1</code>
: Set the the full functionality of the module.

====Verifying the Operation with Wireshark====

Once all the AT Commands have been executed, we should see msg1, msg2, msg3, msg4, RRC logs on the gNB and the registration request from the UE to the AMF on Wireshark.

The screenshot below shows Wireshark running on the CN machine, and shown on the right is the corresponding sequence diagram showing the message exchanges between the UE, gNB, and CN AMF, from the perspective of the CN machine.

[[File:quectel-ue-wireshark-cn.JPG|1000px|center]]

The screenshot below shows Wireshark running on the gNB machine, and shown on the right is the corresponding sequence diagram showing the message exchanges between the UE, gNB, and CN AMF, from the perspective of the gNB machine.

[[File:quectel-ue-wireshark-gnb.JPG|1000px|center]]

====Verifying the Operation with AT Commands====

We run the AT Commands listed below in Minicom, and examine the responses, to verify the operation of the system.

; <code>AT+COPS?</code>
: This output specifies the current operators and their status.
: Expected Output: <code>0,0,"208 92 open cells",11</code>
: Field 1: Availability of Operators, where <code>0</code> specifies Unknown
: Field 2: Operator Selection of operators, where <code>0</code> specifies Automatic
: Field 3: Operator Name, where <code>208 92 Open cell</code> specifies pre-programmed operator name
: Field 4: Access Technology Selected, where <code>11</code> specifies 5G NR connected to a 5G CN

; <code>AT+C5GREG?</code>
: The output of this command shows the 5G network registration status.
: Expected Output: <code>2,1,"1","0",11,16,"01.00007B;00.000000:01.00000C;00.000000"</code>
: Field 1: Enable network registration and location information, where <code>2</code> specifies Unsolicited Mode
: Field 2: Registration Status, where <code>1</code> specifies Registered on Home Network
: Field 3: <code>1</code> specifies a three-byte tracking area code in hexadecimal format
: Field 4: <code>0</code> specifies a five-byte (NR) cell ID in hexadecimal format
: Field 5: <code>11</code> specifies that the connection mode is 5G NR connected to a 5G CN
: Field 6: <code>16</code> specifies the number of octets of the allowed NSSAI information element
: Field 7: <code>01.00007B;00.000000:01.00000C;00.000000</code> specifies the allowed NSSAI

=== UE Scenario 3: COTS Handset ===

The UE can be implemented using the commercial (COTS) handset. This reference architecture will feature the use of the Google Pixel 5A handset. Detailed documentation about this will be added in the near future.

==End-to-End Verification==

===Using <code>ping</code>===

We run <code>ping</code> on the CN machine to verify connectivity from the CN, through the gNB, to the UE. Run the command listed below on the CN machine.  Replace <code>12.1.1.x</code> with the correct IP address of the UE, as assigned by the CN.

    sudo docker exec –it oai-ext-dn ping 12.1.1.x

===Using <code>iperf</code>===

The The The

==Technical Support==

The primary method of technical support is through mailing lists and direct email.

===USRP Mailing List===

The usrp-users mailing list is for discussions specifically involving the USRP hardware and the UHD software.

https://lists.ettus.com/list/usrp-users.lists.ettus.com

The list archives can be found at the link below.

https://lists.ettus.com/empathy/list/usrp-users.lists.ettus.com

===OAI Mailing List===

There are two mailing lists, openair5g-user and openair5g-nr, which are for discussions specifically involving the Open Air Interface (OAI) software stack. More information can be found at the links below.

https://gitlab.eurecom.fr/oai/openairinterface5g/-/wikis/MailingList

https://gitlab.eurecom.fr/oai/openairinterface5g/-/wikis/AskQuestions

===Email===

You can contact the authors about the reference architecture directly via email at <code>support@ettus.com</code>.