Using Ethernet-Based Synchronization on the USRP™ N3xx Devices
Contents
Application Note Number and Authors
AN-158 by Dan Baker, Wan Liu, and Michael Dickens
USRP N3xx Synchronization Options
The USRP N3xx product family supports three different methods of baseband synchronization: external clock and time reference, GPSDO module, and Ethernet-based timing protocol. Using an external clock and time reference source, such as the CDA-2990 accessory, offers a precise and convenient method of baseband synchronization for high channel count systems where devices are colocated, such as in a rackmount configuration. Using the GPSDO module enables synchronization when the devices are physically separated by large distances such as in small cell, RF sensor, TDOA, and distributed testbed applications. However, the GPSDO method typically has more skew than the other two methods and requires line of sight to satellites; indoor, urban, or hostile environments restrict the use of GPSDO. Ethernet-based synchronization enables precise baseband synchronization over large distances in GPS-denied environments. However, this method consumes one of the SFP+ ports of the USRP N3xx devices and therefore reduces the number of connectors available for IQ streaming. This application note provides instructions for synchronizing multiple USRP N3xx devices using the Ethernet-based method.
Ethernet-Based Synchronization Overview
The USRP N3xx product family supports Ethernet-based synchronization using an open source protocol known as White Rabbit. White Rabbit is a fully deterministic Ethernet-based network protocol for general purpose data transfer and synchronization[1]. This project is supported by a collaboration of academic and industry experts such as CERN and GSI Helmholtz Centre for Heavy Ion Research.
White Rabbit is an extension of the IEEE 1588 Precision Time Protocol (PTP) standard, which distributes time references over Ethernet networks. In addition, White Rabbit uses Synchronous Ethernet (SyncE) to distribute a common clock reference over the network across the Ethernet physical layer to ensure frequency syntonization between all nodes. This combination of SyncE and PTP, in addition to further measurements, provides sub-nanosecond synchronization over distances of up to 10 km. The White Rabbit extension of the IEEE 1588-2008 standard is in the final stages of becoming generalized as the IEEE 1588 High Accuracy profile[2].
The USRP N3xx product family implements the White Rabbit protocol using a combination of the FPGA and dedicated clocking resources. The USRP N3xx operates as a "slave" node [this terminology comes from White Rabbit] and a White Rabbit "master" node is required to be networked to the USRP. Safran Timing Solutions (formerly Seven Solutions aka SevenSols) provides White Rabbit hardware that works with the USRP N3xx devices to create synchronous clock and time references that are precisely aligned across all devices in the network. See the “Required Accessories” section for details on the required external hardware. The USRP N3xx devices do not support IQ sample streaming over this protocol. Therefore, only one of SFP+ ports is available for streaming when using White Rabbit synchronization; one can use the QSFP port on the N3xx USRP -- which supports 2x 10 GbE links -- for higher bandwidth applications.
Required Accessories
White Rabbit synchronization utilizes specific optical SFP transceivers and single mode fiber optic cables to achieve precise time alignment, as documented on the project website. The USRP N3xx has been tested to work as a White Rabbit "slave" using the AXGE-1254-0531 SFP transceiver (blue handle), the AXGE-3454-0531 SFP transceiver (purple handle), and a G652 type single mode fiber optic cable. The specifics on actual setup are noted in the next section.
Safran Timing Solutions (formerly Seven Solutions aka SevenSols) is a provider of White Rabbit equipment, including the WR-LEN and the White Rabbit Switch (WRS) noted in the next section. The USRP N3xx was tested to work with both the WR-LEN and the WRS and the AXGE transcivers. All accessories required for White Rabbit operation can be purchased directly from Safran Timing Solutions. The AXGE SFP transceivers and fiber optic cables can also be purchased individually by contacting Safran Timing Solutions.
For more information on White Rabbit accessories, visit the links below:
White Rabbit SFP wiki:
Safran Timing Solutions WR-LEN
Safran Timing Solutions WRS:
NOTE: There are transceivers other than the AXGE that have been tested with the WR-LEN, WRS, and USRP, such as the FS SFP 1000BASE-BX-D BiDi 1490nm-TX/1310nm-RX 10km (purple) and FS SFP 1000BASE-BX-U BiDi 1310nm-TX/1490nm-RX 10km (blue) transceivers for up to 10 km distances via cables such as the FS 1m (3ft) Fiber Patch Cable. The AXGE transceivers are known compatible with pre-existing calibration information in the WRS and WR-LEN. Calibrating any transceivers is beyond the scope of this document.
System Configuration
The White Rabbit feature of the USRP N3xx product family is based on standard networking technology, therefore many system topologies are possible. The USRP N3xx device works as a downstream "slave" node only and must receive its synchronization reference from an upstream "master" node. This section shows examples of typical configurations used to synchronize a network of multiple USRP N3xx devices.
Figure 1 shows a WRS operating as the "master" node connected to several USRP N3xx devices. Note that a "master" SFP port requires the purple SFP transceiver mentioned in the previous section, and a "slave" SFP port requires the blue SFP transceiver. With the WX FPGA bitfile, the USRP N3xx use the SFP+ port 0 for White Rabbit and SFP+ 1 port for IQ streaming.
Download all FPGA images for the version of the USRP Hardware Driver (UHD) installed on the host PC by running the following command in a terminal:
uhd_images_downloader
Load the default WX bitfile by running:
uhd_image_loader --args type=n3xx,mgmt_addr=<DEVICE_IP_ADDRESS>,fpga=WX
or
ssh root@<DEVICE_IP_ADDRESS> "uhd_image_loader --args type=n3xx,mgmt_addr=localhost,fpga=WX"
Note that the XQ FPGA bitfile takes White Rabbit on SFP+ port 0, but uses the QSFP+ port for data links providing for 2x SFP+ at 10 Gb/s each for an aggregate of 20 Gb/s, whereas the WX FPGA bitfile provides for just a single SFP+ data link for a total of 10 Gb/s. Using the QSFP+ port can be accomplished via 2 primary ways: (1) a breakout cable from QSFP+ <-> 4x SFP+ (or QSFP28 <-> 4x SFP28) and then a NIC providing 2 SFP+ ports (such as the Nvidia Mellanox Connect-X 4 or Intel X710-DA2) or a QSFP+ <-> QSPF+ cable and a NIC providing a QSFP+ port (such as the Intel XL710-QDA2).
Using the UHD API, configure the USRP application to use “internal” clock source and “sfp0” time source:
usrp->set_clock_source("internal")
usrp->set_time_source("sfp0")
UHD utilities can also do these settings, for example:
uhd_usrp_probe --args type=n3xx,mgmt_addr=<DEVICE_IP_ADDRESS>,time_source=sfp0,clock_source=internal
The White Rabbit IP running on the FPGA disciplines the internal VCXO of the USRP N3xx to the clock reference from the upstream "master" node in the network. See the USRP N3xx block diagram for reference.
The WRS/WR-LEN device needs to be configured as "master" on the ports connected to the USRP N3xx devices. Users can make this configuration with the WR-GUI application provided by Safran Time Solutions or with a serial console connection to the WRS/WR-LEN device. See the WRS/WR-LEN manual for detailed instructions. After White Rabbit lock is achieved, the standard USRP N3xx synchronization process completes and the devices are ready for use.
In addition to operating as a "master", the WRS and WR-LEN devices can operate as a "grandmaster" by receiving clock and time references from an external source. This feature is useful for situations where the entire White Rabbit network needs to be disciplined to GPS or other high accuracy synchronization equipment such as a rubidium source. See the WRS/WR-LEN documentation for more information on "grandmaster" mode.
Synchronization Example
This section provides an example measurement of the timing alignment between multiple USRP N3xx devices synchronized using White Rabbit, with varying fiber cable lengths. As shown in Figure 3, a White Rabbit Switch in master mode is connected to one USRP N3xx device using a 5 km spool of fiber, and to another USRP N3xx device using 1 m of fiber. The synchronization performance was measured by probing the exported PPS signal, which is in the sample clock domain on both USRP N3xx devices thereby demonstrating sample clock and timestamp alignment. The time difference between each PPS edge was measured with an oscilloscope at room temperature in a laboratory environment. As shown in Figure 4, the resulting measurement shows about 222 ps of skew between the two USRP N3xx devices, thereby demonstrating the sub-nanosecond synchronization of White Rabbit over long distances.
The frequency accuracy of the internal oscillator of each USRP N3xx slave node is derived from the frequency accuracy of the upstream master node, in a manner similar to disciplining to an external clock reference source connected to the REF IN port. By connecting a high accuracy frequency source such as a rubidium reference to the master White Rabbit device in grandmaster mode, all USRP N3xx devices in the White Rabbit network would inherit this frequency accuracy.
Checking USRP for White Rabbit lock
There are 3 primary ways to check the status of White Rabbit connectivity on the USRP.
N3XX Built-In Self Test
After logging into the USRP and while one of the White Rabbit FPGA images is loaded, execute the following command:
n3xx_bist --skip-fpga-reload -v whiterabbit
If the USRP has White Rabbit lock, then it will return the following after around 5 seconds of command issue:
Executing test method: bist_whiterabbit
{
"whiterabbit": {
"error_msg": "",
"lock_status": 1,
"status": true
}
}
BIST complete.
If there is no White Rabbit lock, then the test will run for roughly 45 seconds and return:
Executing test method: bist_whiterabbit
{
"whiterabbit": {
"error_msg": "",
"lock_status": 0,
"status": false
}
}
BIST complete.
Accessing the White Rabbit FPGA Core
All White Rabbit devices provide a core that can be logged into for the purposes of viewing information and setting calibration parameters (among other things). In UHD, the White Rabbit core is integrated into the FPGA image, and is made available from the USRP's login as a Linux TTY device. To access the White Rabbit core, from the USRP's login issue the following command:
screen /dev/ttyUL0 115200
and then hit the return key to get the wrc# prompt. Issue the command help for a list of command available within the White Rabbit core. For example, the gui command will show the status of the device via refreshing text-based "window". When the USRP has White Rabbit lock, the window will show something like the following
WR PTP Core Sync Monitor wrpc-v4.2-2-g97f418e Esc = exit TAI Time: Thu, Feb 5, 1970, 00:30:52 Link status: wru1: Link up (RX: 4350, TX: 2475) IPv4: BOOTP running Mode: WR Slave Locked Calibrated PTP status: slave Synchronization status: Servo state: TRACK_PHASE Phase tracking: ON Aux clock 0 status: enabled Timing parameters: Round-trip time (mu): 907377 ps Master-slave delay: 452548 ps Master PHY delays: TX: 224037 ps, RX: 227977 ps Slave PHY delays: TX: 0 ps, RX: 1600 ps Total link asymmetry: 2281 ps Cable rtt delay: 453763 ps Clock offset: 1 ps Phase setpoint: 3848 ps Skew: 1 ps Update counter: 737
whereas if there is no White Rabbit lock, the window will show something like
WR PTP Core Sync Monitor wrpc-v4.2-2-g97f418e Esc = exit TAI Time: Thu, Jan 1, 1970, 00:12:39 Link status: wru1: Link up (RX: 760, TX: 1162) IPv4: BOOTP running Mode: WR Off PTP status: listening Sync info not valid
NOTE: Viewing information to see the status of the White Rabbit core is recommended when debugging White Rabbit connectivity. Changing any parameters within the White Rabbit FPGA core is an advanced topic that is beyond the scope of this document, and done at your own risk.
Running benchmark_rate
Assuming that the prior 2 lock tests work, then this test shows that UHD is working with White Rabbit signals by executing actual data transfer via the benchmark_rate utility. Note that this utility does not verify data integrity nor data phase any synchronization.
For 2 USRPs, here is an example benchmark_rate command showing addr# for both data and management, as well as basic settings for White Rabbit:
/usr/local/lib/uhd/examples/benchmark_rate --args type=n3xx,mgmt_addr0=10.9.8.202,addr0=10.9.10.2,mgmt_addr1=10.9.8.203,addr1=10.9.20.2,master_clock_rate=250e6,time_source=sfp0,clock_source=internal --channel 0,1,2,3 --rx_rate 25e6 --tx_rate 25e6 --multi_streamer
If there is no White Rabbit lock, then this command will error out showing something like the following:
[ERROR] [RPC] Failed to lock SFP timebase. [ERROR] [MPM.PeriphManager] sfp0 timebase failed to lock within 40 seconds. Status: 0x0 [ERROR] [MPM.RPCServer] init() failed with error: Failed to lock SFP timebase. Error: RuntimeError: Error during RPC call to `init'. Error message: Failed to lock SFP timebase.
If there is White Rabbit lock, then if this command fails any issue should not be related to White Rabbit -- most likely networking issues.
