View source for Using Ethernet-Based Synchronization on the USRP™ N3xx Devices

== Application Note Number and Authors ==

'''AN-158''' by Dan Baker, Wan Liu, and Michael Dickens
<!--
This is commented out for rendering, but do keep it updated.
== Revision History ==
{| class="wikitable"
!Date
!Author
!Details
|-
|style="text-align:center;"| 2018-05-01 
|style="text-align:left;"| Dan Baker, Wan Liu
|style="text-align:center;"| Initial creation
|-
|style="text-align:center;"| 2023-04-11 
|style="text-align:left;"| Michael Dickens
|style="text-align:center;"| Add NOTE about transceiver compatibility and calibration
|-
|style="text-align:center;"| 2023-04-19
|style="text-align:left;"| Michael Dickens
|style="text-align:center;"| Add info about WR core access & lock testing
|}
-->

== USRP N3xx Synchronization Options ==

The USRP N3xx product family supports three different methods of baseband synchronization: external clock and time reference, GPSDO module, and Ethernet-based timing protocol.  Using an external clock and time reference source, such as the [https://www.ettus.com/product/details/OctoClock-G CDA-2990] accessory, offers a precise and convenient method of baseband synchronization for high channel count systems where devices are colocated, such as in a rackmount configuration. Using the GPSDO module enables synchronization when the devices are physically separated by large distances such as in small cell, RF sensor, TDOA, and distributed testbed applications. However, the GPSDO method typically has more skew than the other two methods and requires line of sight to satellites; indoor, urban, or hostile environments restrict the use of GPSDO. Ethernet-based synchronization enables precise baseband synchronization over large distances in GPS-denied environments. However, this method consumes one of the SFP+ ports of the USRP N3xx devices and therefore reduces the number of connectors available for IQ streaming. This application note provides instructions for synchronizing multiple USRP N3xx devices using the Ethernet-based method.

== Ethernet-Based Synchronization Overview ==

The USRP N3xx product family supports Ethernet-based synchronization using an open source protocol known as [https://www.ohwr.org/projects/white-rabbit/wiki White Rabbit].  White Rabbit is a fully deterministic Ethernet-based network protocol for general purpose data transfer and synchronization[1]. This project is supported by a collaboration of academic and industry experts such as CERN and GSI Helmholtz Centre for Heavy Ion Research.

White Rabbit is an extension of the [https://www.ohwr.org/projects/wr-std/wiki IEEE 1588 Precision Time Protocol (PTP) standard], which distributes time references over Ethernet networks. In addition, White Rabbit uses Synchronous Ethernet (SyncE) to distribute a common clock reference over the network across the Ethernet physical layer to ensure frequency syntonization between all nodes. This combination of SyncE and PTP, in addition to further measurements, provides sub-nanosecond synchronization over distances of up to 10 km. The White Rabbit extension of the IEEE 1588-2008 standard is in the final stages of becoming generalized as the IEEE 1588 High Accuracy profile[2]. 

The USRP N3xx product family implements the White Rabbit protocol using a combination of the FPGA and dedicated clocking resources.  The USRP N3xx operates as a "slave" node [this terminology comes from White Rabbit] and a White Rabbit "master" node is required to be networked to the USRP. Safran Timing Solutions (formerly Seven Solutions aka SevenSols) provides White Rabbit hardware that works with the USRP N3xx devices to create synchronous clock and time references that are precisely aligned across all devices in the network. See the “Required Accessories” section for details on the required external hardware. The USRP N3xx devices do not support IQ sample streaming over this protocol. Therefore, only one of SFP+ ports is available for streaming when using White Rabbit synchronization; one can use the QSFP port on the N3xx USRP -- which supports 2x 10 GbE links -- for higher bandwidth applications.

== Required Accessories ==

White Rabbit synchronization utilizes specific optical SFP transceivers and single mode fiber optic cables to achieve precise time alignment, as documented on the project website. The USRP N3xx has been tested to work as a White Rabbit "slave" using the AXGE-1254-0531 SFP transceiver (blue handle), the AXGE-3454-0531 SFP transceiver (purple handle), and a G652 type single mode fiber optic cable. The specifics on actual setup are noted in the next section.

Safran Timing Solutions (formerly Seven Solutions aka SevenSols) is a provider of White Rabbit equipment, including the WR-LEN and the White Rabbit Switch (WRS) noted in the next section.  The USRP N3xx was tested to work with both the WR-LEN and the WRS and the AXGE transcivers. All accessories required for White Rabbit operation can be purchased directly from Safran Timing Solutions. The AXGE SFP transceivers and fiber optic cables can also be purchased individually by contacting Safran Timing Solutions.

For more information on White Rabbit accessories, visit the links below:

White Rabbit SFP wiki:

* https://www.ohwr.org/projects/white-rabbit/wiki/SFP 

Safran Timing Solutions WR-LEN

* https://safran-navigation-timing.com/product/white-rabbit-len/

Safran Timing Solutions WRS:

* https://safran-navigation-timing.com/product/white-rabbit-switch-v3-4/

NOTE: There are transceivers other than the AXGE that have been tested with the WR-LEN, WRS, and USRP, such as the [https://www.fs.com/products/75336.html?attribute=10364&id=199842 FS SFP 1000BASE-BX-D BiDi 1490nm-TX/1310nm-RX 10km] (purple) and [https://www.fs.com/products/75335.html?attribute=96223&id=3772184 FS SFP 1000BASE-BX-U BiDi 1310nm-TX/1490nm-RX 10km] (blue) transceivers for up to 10 km distances via cables such as the [https://www.fs.com/products/40446.html FS 1m (3ft) Fiber Patch Cable]. The AXGE transceivers are known compatible with pre-existing calibration information in the WRS and WR-LEN. Calibrating any transceivers is beyond the scope of this document.

== System Configuration ==

The White Rabbit feature of the USRP N3xx product family is based on standard networking technology, therefore many system topologies are possible. The USRP N3xx device works as a downstream "slave" node only and must receive its synchronization reference from an upstream "master" node. This section shows examples of typical configurations used to synchronize a network of multiple USRP N3xx devices.

Figure 1 shows a WRS operating as the "master" node connected to several USRP N3xx devices. Note that a "master" SFP port requires the '''purple''' SFP transceiver mentioned in the previous section, and a "slave" SFP port requires the '''blue''' SFP transceiver. With the WX FPGA bitfile, the USRP N3xx use the SFP+ port 0 for White Rabbit and SFP+ 1 port for IQ streaming.

Download all FPGA images for the version of the USRP Hardware Driver (UHD) installed on the host PC by running the following command in a terminal:

    uhd_images_downloader

Load the [https://files.ettus.com/manual/page_usrp_n3xx.html#n3xx_rh_sfp_protocols default WX bitfile] by running:

    uhd_image_loader --args type=n3xx,mgmt_addr=<DEVICE_IP_ADDRESS>,fpga=WX

or

    ssh root@<DEVICE_IP_ADDRESS> "uhd_image_loader --args type=n3xx,mgmt_addr=localhost,fpga=WX"

Note that the XQ FPGA bitfile takes White Rabbit on SFP+ port 0, but uses the QSFP+ port for data links providing for 2x SFP+ at 10 Gb/s each for an aggregate of 20 Gb/s, whereas the WX FPGA bitfile provides for just a single SFP+ data link for a total of 10 Gb/s. Using the QSFP+ port can be accomplished via 2 primary ways: (1) a breakout cable from QSFP+ <-> 4x SFP+ (or QSFP28 <-> 4x SFP28) and then a NIC providing 2 SFP+ ports (such as the Nvidia Mellanox Connect-X 4 or Intel X710-DA2) or a QSFP+ <-> QSPF+ cable and a NIC providing a QSFP+ port (such as the Intel XL710-QDA2).

Using the UHD API, configure the USRP application to use “internal” clock source and “sfp0” time source:

    usrp->set_clock_source("internal")
    usrp->set_time_source("sfp0")

UHD utilities can also do these settings, for example:

    uhd_usrp_probe --args type=n3xx,mgmt_addr=<DEVICE_IP_ADDRESS>,time_source=sfp0,clock_source=internal

The White Rabbit IP running on the FPGA disciplines the internal VCXO of the USRP N3xx to the clock reference from the upstream "master" node in the network. See the [https://www.ettus.com/content/images/USRP_N310_MB_Diagram.png USRP N3xx block diagram] for reference.

The WRS/WR-LEN device needs to be configured as "master" on the ports connected to the USRP N3xx devices. Users can make this configuration with the WR-GUI application provided by Safran Time Solutions or with a serial console connection to the WRS/WR-LEN device. See the WRS/WR-LEN manual for detailed instructions.  After White Rabbit lock is achieved, the standard USRP N3xx synchronization process completes and the devices are ready for use.

[[File:white_rabbit_usrp_1.png|700px|center]]
<center>Figure 1.  USRP N3xx synchronization using a White Rabbit Switch</center>

In addition to operating as a "master", the WRS and WR-LEN devices can operate as a "grandmaster" by receiving clock and time references from an external source.  This feature is useful for situations where the entire White Rabbit network needs to be disciplined to GPS or other high accuracy synchronization equipment such as a rubidium source. See the WRS/WR-LEN documentation for more information on "grandmaster" mode.

[[File:white_rabbit_usrp_2.png|700px|center]]
<center>Figure 2 shows multiple WRS devices cascaded to expand the network beyond 18 USRP N3xx devices</center>

== Synchronization Examples ==

White Rabbit protocol is designed with multiple tiers and endpoints coming from different tiers in mind. This section provides 2 example measurements of the timing alignment between 2 USRP N3xx devices synchronized using White Rabbit, with varying fiber cable lengths.

As shown in Figure 3, a White Rabbit Switch in "master" mode is connected to one USRP N3xx device using a 5 km spool of fiber, and to another USRP N3xx device using 1 m of fiber.  The synchronization performance was measured by probing the PPS signal exported on the TRIG OUT port, which is in the sample clock domain on both USRP N3xx devices thereby demonstrating sample clock and timestamp alignment.  The time difference between each PPS edge was measured with an oscilloscope at room temperature in a laboratory environment.  As shown in Figure 4, the resulting measurement shows about 222 ps of skew between the two USRP N3xx devices, thereby demonstrating the sub-nanosecond synchronization of White Rabbit over long distances.

As shown in Figure 5, a White Rabbit Switch in "master" mode is connected to two WR-LEN devices, each configured in "slave_wr1" mode where port 1 is "master" and port 2 is "slave". Each WR-LEN is then connected to a USRP, and measurement repeated. As shown in Figure 6, the resulting measurement shows about 222 ps of skew between the two USRP N3xx devices.

The frequency accuracy of the internal oscillator of each USRP N3xx "slave" node is derived from the frequency accuracy of the upstream "master" node in a manner similar to disciplining to an external clock reference source connected to the REF IN port.  By connecting a high accuracy frequency source such as a rubidium reference to the furthest upstream White Rabbit device in "grandmaster" mode, all USRP N3xx devices in the White Rabbit network would inherit this frequency accuracy.  



[[File:white_rabbit_usrp_3.png|700px|center]]
<center>Figure 3.  Test setup to measure the alignment of two same-tier USRP N3xx devices separated by 5 km of fiber</center>



[[File:white_rabbit_usrp_4.png|700px|center]]
<center>Figure 4.  USRP N3xx measured skew of about 218 ps using 5 km of fiber</center>



[[File:white_rabbit_usrp_5.png|700px|center]]
<center>Figure 5.  Test setup to measure the alignment of two multi-tier USRP N3xx devices separated by 5 km of fiber</center>



[[File:white_rabbit_usrp_6.png|700px|center]]
<center>Figure 6.  Example showing USRP N3xx measured skew of about 73 ps using 5 km of fiber</center>

== Checking USRP for White Rabbit lock ==

There are 3 primary ways to check the status of White Rabbit connectivity on the USRP.

=== N3XX Built-In Self Test ===

After logging into the USRP and while one of the White Rabbit FPGA images is loaded, execute the following command:

    n3xx_bist --skip-fpga-reload -v whiterabbit

If the USRP has White Rabbit lock, then it will return the following after around 5 seconds of command issue:

    Executing test method: bist_whiterabbit
    {
        "whiterabbit": {
             "error_msg": "",
             "lock_status": 1,
             "status": true
        }
    }
    BIST complete.

If there is no White Rabbit lock, then the test will run for roughly 45 seconds and return:

    Executing test method: bist_whiterabbit
    {
        "whiterabbit": {
             "error_msg": "",
             "lock_status": 0,
             "status": false
        }
    }
    BIST complete.

=== Accessing the White Rabbit FPGA Core ===

All White Rabbit devices provide a core that can be logged into for the purposes of viewing information and setting calibration parameters (among other things). In UHD, the White Rabbit core is integrated into the FPGA image, and is made available from the USRP's login as a Linux TTY device. To access the White Rabbit core, from the USRP's login issue the following command:

    screen /dev/ttyUL0 115200

and then hit the <code>return</code> key to get the <code>wrc#</code> prompt. Issue the command <code>help</code> for a list of command available within the White Rabbit core. For example, the <code>gui</code> command will show the status of the device via refreshing text-based "window". When the USRP has White Rabbit lock, the window will show something like the following

    WR PTP Core Sync Monitor wrpc-v4.2-2-g97f418e
    Esc = exit
    
    TAI Time:                  Thu, Feb 5, 1970, 00:30:52
    
    Link status:
    wru1: Link up   (RX: 4350, TX: 2475) IPv4: BOOTP running
    Mode: WR Slave   Locked Calibrated
    
    PTP status: slave
    
    Synchronization status:
    Servo state:               TRACK_PHASE
    Phase tracking:            ON
    Aux clock 0 status:        enabled
    
    Timing parameters:
    Round-trip time (mu):               907377 ps
    Master-slave delay:                 452548 ps
    Master PHY delays:           TX:    224037 ps, RX:    227977 ps
    Slave PHY delays:            TX:         0 ps, RX:      1600 ps
    Total link asymmetry:                 2281 ps
    Cable rtt delay:                    453763 ps
    Clock offset:                            1 ps
    Phase setpoint:                       3848 ps
    Skew:                                    1 ps
    Update counter:                        737

whereas if there is no White Rabbit lock, the window will show something like

    WR PTP Core Sync Monitor wrpc-v4.2-2-g97f418e
    Esc = exit
    
    TAI Time:                  Thu, Jan 1, 1970, 00:12:39
    
    Link status:
    wru1: Link up   (RX: 760, TX: 1162) IPv4: BOOTP running
    Mode: WR Off
    
    PTP status: listening
    
    Sync info not valid

NOTE: Viewing information to see the status of the White Rabbit core is recommended when debugging White Rabbit connectivity. Changing any parameters within the White Rabbit FPGA core is an advanced topic that is beyond the scope of this document, and done at your own risk.

=== Running <code>benchmark_rate</code> ===

Assuming that the prior 2 lock tests work, then this test shows that UHD is working with White Rabbit signals by executing actual data transfer via the <code>benchmark_rate</code> utility. Note that this utility does not verify data integrity nor data phase any synchronization.

For 2 USRPs, here is an example <code>benchmark_rate</code> command showing <code>addr#</code> for both data and management, as well as basic settings for White Rabbit:

    /usr/local/lib/uhd/examples/benchmark_rate --args type=n3xx,mgmt_addr0=10.9.8.202,addr0=10.9.10.2,mgmt_addr1=10.9.8.203,addr1=10.9.20.2,master_clock_rate=250e6,time_source=sfp0,clock_source=internal --channel 0,1,2,3 --rx_rate 25e6 --tx_rate 25e6 --multi_streamer

If there is no White Rabbit lock, then this command will error out showing something like the following:

    [ERROR] [RPC] Failed to lock SFP timebase.
    [ERROR] [MPM.PeriphManager] sfp0 timebase failed to lock within 40 seconds. Status: 0x0
    [ERROR] [MPM.RPCServer] init() failed with error: Failed to lock SFP timebase.
    Error: RuntimeError: Error during RPC call to `init'. Error message: Failed to lock SFP timebase.

If there is White Rabbit lock, then if this command fails any issue should not be related to White Rabbit -- most likely networking issues.

== References ==

* [1] https://www.ohwr.org/projects/white-rabbit 
* [2] https://www.ohwr.org/projects/wr-std/wiki 

[[Category:Application Notes]]