https://kb.ettus.com/api.php?action=feedcontributions&user=SamReiter&feedformat=atomEttus Knowledge Base - User contributions [en]2024-03-29T14:36:15ZUser contributionsMediaWiki 1.26.2https://kb.ettus.com/index.php?title=WBX&diff=4883WBX2020-03-26T23:05:51Z<p>SamReiter: /* Gains */</p>
<hr />
<div>== Device Overview ==<br />
The WBX is a wide bandwidth transceiver that provides up to 100 mW of output power and a noise figure of 5 dB. The LO's for the receive and transmit chains operate independently. The WBX daughterboard is supported by the USRP Hardware Driver™ (UHD) software API for seamless integration into existing applications.<br />
<br />
The WBX provides phase coherent operation, although with a 180-degree ambiguity, which must be calibrated out in the application. For phase-coherent applications, Ettus Research recommends the SBX with the N200/N210 or the SBX or UBX with the X300/X310.<br />
<br />
== Key Features==<br />
{|<br />
|style="vertical-align:top"|<br />
* Frequency Range: 50MHz - 2.2GHz<br />
* Versions: 40MHz / 120MHz<br />
* Power Output: 100mW<br />
* Noise Figure: 5dB<br />
|[[File:Product wbx 40.jpg|250px|center]] <br />
|[[File:Product wbx 120.jpg|250px|center]] <br />
|}<br />
==Daughterboard Specifications==<br />
===Features===<br />
* 2 quadrature frontends (1 transmit, 1 receive)<br />
** Defaults to direct conversion<br />
** Can be used in low IF mode through lo_offset with uhd::tune_request_t<br />
* Independent receive and transmit LO's and synthesizers<br />
** Allows for full-duplex operation on different transmit and receive frequencies<br />
** Can be set to use Integer-N tuning for better spur performance with uhd::tune_request_t<br />
<br />
===Antennas===<br />
Transmit: '''TX/RX'''<br />
<br />
Receive: '''TX/RX''' or '''RX2'''<br />
* '''Frontend 0:''' Complex baseband signal for selected antenna<br />
* '''Note:''' The user may set the receive antenna to be TX/RX or RX2. However, when using a WBX board in full-duplex mode, the receive antenna will always be set to RX2, regardless of the settings.<br />
<br />
===Gains===<br />
* Transmit Gains: '''PGA0''', Range: 0-31.5dB<br />
* Receive Gains: '''PGA0''', Range: 0-31.5dB<br />
<br />
===Bandwidths===<br />
* WBX: 40 MHz, RX & TX<br />
* WBX-120: 120 MHz, RX & TX<br />
<br />
===Sensors===<br />
* '''lo_locked:''' boolean for LO lock state<br />
<br />
==RF Specifications==<br />
===Freq Range===<br />
* 50MHz - 2.2GHz<br />
<br />
===Noise Figure===<br />
* 2 - 4 dB @ (50MHz ~ 1.2GHz) <br />
* 4 - 8 dB @ (1.2GHz ~ 2.2GHz)<br />
<br />
===RX IIP3 (Max)=== <br />
* 10 - 18 dBm<br />
<br />
===RX IQ Imbalance===<br />
* -30 dBc<br />
<br />
===TX Power (Max)===<br />
* 18 - 20 dBm @ (50MHz ~ 1.4 GHz) <br />
* 12 - 18dBm @ (1.4GHz ~ 2.2 GHz)<br />
<br />
===TX OIP3===<br />
* 30 - 32 dBm @ (50MHz ~ 800MHz) <br />
* 25 - 30 dBm @ (800MHz ~ 2.2GHz)<br />
<br />
===TX IQ Imbalance===<br />
* -30 dBc @ (50MHz ~ 1.9GHz) <br />
*- 24 dBc @ (1.9GHz ~ 2.2GHz)<br />
<br />
===Input/Output Impedance===<br />
* All RF Ports are matched to 50 Ohm with -10dB or better return loss generally. Detailed test is pending.<br />
<br />
===Input Power Levels===<br />
* The maximum input power for the WBX is -15 dBm.<br />
<br />
==RF Performance Data==<br />
* [http://files.ettus.com/performance_data/wbx/WBX-without-UHD-corrections.pdf WBX without UHD Corrections]<br />
<br />
==Hardware Specifications==<br />
* Ettus Research recommends to always use the latest stable version of UHD<br />
<br />
===WBX-40===<br />
* Current Hardware Revision: 1<br />
* Minimum version of UHD required: 3.8.0<br />
<br />
===WBX-120===<br />
* Current Hardware Revision: 1<br />
* Minimum version of UHD required: 3.8.0<br />
<br />
==Environmental Specifications==<br />
===Operating Temperature Range===<br />
* 0-40 °C<br />
<br />
===Operating Humidity Range===<br />
* 10% to 90% non-condensing<br />
<br />
==USRP Compatibility==<br />
===WBX-40===<br />
* N or X Series<br />
<br />
===WBX-120===<br />
* X Series only<br />
<br />
==Phase Synchronization==<br />
The WBX daughterboard is capable of phase-synchronous operation, but with a with a 180-degree ambiguity, which must be calibrated out in the user application. The SBX, UBX, TwinRX daughterboards are recommended for phase-coherent applications.<br />
<br />
==Schematics==<br />
===WBX===<br />
[http://files.ettus.com/schematics/wbx/WBX.pdf WBX Schematics]<br />
<br />
[http://files.ettus.com/schematics/wbx/wbx_fe.pdf WBX FE Schematics]<br />
<br />
==Key Component Datasheets==<br />
{| class="wikitable" style="width:80%"<br />
!Part Number<br />
!Description<br />
!Schematic ID (Page)<br />
|-<br />
<br />
|[http://www.analog.com/media/en/technical-documentation/data-sheets/ADA4937-1_4937-2.pdf ADA4937]<br />
|Ultralow Distortion Differential ADC Driver<br />
|U304 (2)<br />
|-<br />
<br />
|[http://www.analog.com/media/en/technical-documentation/data-sheets/ADP3336.pdf ADP3336]<br />
|High Accuracy Ultralow IQ, 500 mA anyCAP® Adjustable Low Dropout Regulator<br />
|U306, U308 (2); U503, U505 (4)<br />
|-<br />
<br />
|[http://www.analog.com/media/en/technical-documentation/evaluation-documentation/ADL5387.pdf ADL5387]<br />
|50 MHz to 2 GHz Quadrature Demodulator<br />
|U307 (2)<br />
|-<br />
<br />
|[http://www.analog.com/media/en/technical-documentation/data-sheets/ADL5385.pdf ADL5385]<br />
|50 MHz to 2200 MHz Quadrature Modulator<br />
|U501 (4,5)<br />
|-<br />
<br />
|[http://www.analog.com/media/en/technical-documentation/data-sheets/ADF4350.pdf ADF4350] / [http://www.analog.com/media/en/technical-documentation/data-sheets/ADF4351.pdf ADF4351]<br />
|Wideband Synthesizer with Integrated VCO<br />
|U201 (3); U401 (5)<br />
|-<br />
<br />
|[http://media.digikey.com/pdf/Data%20Sheets/Analog%20Devices%20PDFs/HMC472LP4.pdf HMC472LP4]<br />
|Attenuator <br />
|U302 (2); U504 (4,5)<br />
|-<br />
<br />
|[http://www.avagotech.com/docs/AV02-1985EN MGA82563]<br />
|Amplifier<br />
|U313 (2)<br />
|-<br />
<br />
|[http://ww1.microchip.com/downloads/en/DeviceDoc/21210G.pdf 24LC024]<br />
|EEPROM<br />
|U202 (3); U403 (5)<br />
|-<br />
<br />
|[https://www.minicircuits.com/pdfs/GVA-84+.pdf GVA−84+]<br />
|Amplifier<br />
|U502 (4)<br />
|-<br />
|}<br />
<br />
==Mechanical Information==<br />
===Drawings===<br />
* [[Media:cu ettus-wbx-cca.pdf]]<br />
<br />
==RF Connectors==<br />
* The WBX daughterboard features female MCX connectors for both the TX/RX and RX2 connectors.<br />
<br />
==Certifications==<br />
===RoHS===<br />
As of December 1st, 2010 all Ettus Research products are RoHS compliant unless otherwise noted. More information can be found at [http://ettus.com/legal/rohs-information http://ettus.com/legal/rohs-information]<br />
<br />
===China RoHS=== <br />
'''Management Methods for Controlling Pollution Caused by Electronic Information Products Regulation'''<br />
<br />
'''Chinese Customers''' <br />
<br />
National Instruments is in compliance with the Chinese policy on the Restriction of Hazardous Substances (RoHS) used in Electronic Information Products. For more information about the National Instruments China RoHS compliance, visit [http://www.ni.com/environment/rohs_china ni.com/environment/rohs_china].<br />
<br />
==Certificate of Volatility==<br />
===WBX-40/WBX-120===<br />
* [[Media:volatility UBX CBX WBX SBX r1 1.pdf]]<br />
<br />
==Downloads==<br />
<br />
[http://files.ettus.com/manual/md_fpga.html FPGA Resources]<br />
<br />
[http://files.ettus.com/binaries/uhd_stable/ UHD Stable Binaries]<br />
<br />
[https://github.com/EttusResearch/uhd UHD Source Code on Github]<br />
<br />
<br />
[[Category:Hardware Resources]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4881Synchronizing USRP Events Using Timed Commands in UHD2020-03-06T18:38:36Z<p>SamReiter: /* A System-Level Example */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-883'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-03-1 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 0.1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
For more detail, see [https://files.ettus.com/manual/page_sync.html USRP Manual: Device Synchronization]<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; // only the bottom 4 lines: 0xF = 00001111 = Pin 0, 1, 2, 3<br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.00<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps(); // define t=0<br />
<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // reset HIGH (async)<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=2<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // set HIGH @ t=4<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=6<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4880Synchronizing USRP Events Using Timed Commands in UHD2020-03-06T18:34:45Z<p>SamReiter: /* A System-Level Example */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-883'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-03-1 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 0.1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; // only the bottom 4 lines: 0xF = 00001111 = Pin 0, 1, 2, 3<br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.00<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps(); // define t=0<br />
<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // reset HIGH (async)<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=2<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // set HIGH @ t=4<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=6<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Application_Notes&diff=4879Application Notes2020-03-02T03:59:16Z<p>SamReiter: </p>
<hr />
<div>Application Notes (AN) and technical articles written by engineers, for engineers. These articles offer experienced analysis, design ideas, reference designs, and tutorials—to make you productive and successful using USRP devices.<br />
<br />
{| class="wikitable"<br />
!colspan="4"|Application Notes<br />
|-<br />
<br />
<br />
! style="text-align:center;"| Number<br />
! style="text-align:center;"| Title<br />
! style="text-align:center;"| Abstract<br />
! style="text-align:center;"| Author<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-445<br />
|style="width: 30%;"| [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on Linux]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Linux platform. Other alternate installation methods are also discussed.<br />
|style="width: 10%; text-align: center;"| Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-788<br />
|style="width: 30%;"| [[Building and Installing the USRP Open-Source Toolchain (UHD and GNU Radio) on OS X]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Mac OS X platform.<br />
|style="width: 10%; text-align: center;"|Michael Dickens<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-611<br />
|style="width: 30%;"| [[Building and Installing the USRP Open Source Toolchain (UHD and GNU Radio) on Windows]]<br />
|style="width: 50%;"| This AN provides a comprehensive step-by-step guide for building, installing, and maintaining the open-source toolchain, specifically UHD and GNU Radio, for the USRP from source code on the Windows platform.<br />
|style="width: 10%; text-align: center;"|Derek Kozel<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-936<br />
|style="width: 30%;"| [[Verifying the Operation of the USRP Using UHD and GNU Radio]]<br />
|style="width: 50%;"| This AN explains how to use UHD and GNU Radio, once installed, to verify the correct operation of the USRP. Several test procedures are explained in detail. Several tests make use of an optional spectrum analyzer and signal generator.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-561<br />
|style="width: 30%;"| [[Implementation of a Simple FM Receiver in GNU Radio]]<br />
|style="width: 50%;"| This AN shows a quick and simple implementation of an FM receiver for the USRP using GNU Radio. The goal is to easily demonstrate a practical application, and to verify that the USRP is functioning properly.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-188<br />
|style="width: 30%;"| [[Interrogating Passive Wireless SAW Sensors with the USRP]]<br />
|style="width: 50%;"| Typical interrogator design for wireless SAW sensor systems require many discrete components and lengthy build times, making it difficult to rapidly adapt to sensor designs in a research environment. We have employed the USRP B200 as a SAW sensor interrogation system. Interrogation of wideband orthogonal frequency coded (OFC) SAW sensors imposes strict requirements on the timing and synchronization of the transceiver. The USRP FPGA has been modified to operate in a synchronous, pulsed mode of operation, allowing rapid data acquisition and the full 56MHz bandwidth to be utilized. Data from the USRP is passed to a custom matched filter correlator routine to extract sensor parameters. The system is capable of interrogating multiple sensors, simultaneously. Demonstration of the system is accomplished by wirelessly interrogating SAW sensors at 915MHz and extracting temperature.<br />
|style="width: 10%; text-align: center;"|Trip Humphries<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-322<br />
|style="width: 30%;"| [[Experiments with the UBX Daughterboard in the HF Band]]<br />
|style="width: 50%;"| We show the results of experiments with the UBX daughtercard on an USRP X310 platform for use in the HF frequency range, from 1.8MHz to 30MHz. While the UBX is nominally rated for use only down to 10 MHz, with careful flow-graph design, and pre-filtering, it provides quite-good performance across the HF bands.<br />
|style="width: 10%; text-align: center;"|Marcus Leech<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-363<br />
|style="width: 30%;"| [[Implementation of an ADS-B/Mode-S Receiver in GNU Radio]]<br />
|style="width: 50%;"| This AN guides the reader through the implementation of an ADS-B receiver using the gr-air-modes Out-of-Tree (OOT) module for GNU Radio. An explanation of ADS-B is also provided, and several real-world, over-the-air examples and profiled.<br />
|style="width: 10%; text-align: center;"|Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-177<br />
|style="width: 30%;"| [[About USRP Bandwidths and Sampling Rates]]<br />
|style="width: 50%;"| This AN provides insight into the topics of USRP architecture, system bandwidth, host interface throughput, and available sampling rates.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-881<br />
|style="width: 30%;"| [[Selecting a USRP Device]]<br />
|style="width: 50%;"| This AN explores the USRP family at a high level, compares devices across several primary features, and walks the reader through the process of selecting a particular device for the their application.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-492<br />
|style="width: 30%;"| [[Selecting a RF Daughterboard]]<br />
|style="width: 50%;"| This AN explores the RF daughterboards used by the N-series and X-series USRP devices at a high level, compares devices across several primary features, and walks the reader through the process of selecting a particular device for the their application.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-204<br />
|style="width: 30%;"| [[Getting Started with UHD and C++]]<br />
|style="width: 50%;"| This AN explains how to write and build C++ programs that use the UHD API and introduces<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-117<br />
|style="width: 30%;"| [[GPSDO Selection Guide]]<br />
|style="width: 50%;"| This AN explains how to select and use a GPSDO with the USRP B-, N-, and X-series devices.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-503<br />
|style="width: 30%;"| [[Converting an X310 into an NI-USRP Rio]]<br />
|style="width: 50%;"| This Application Note explains how to use an Ettus Research-branded USRP with LabVIEW, and in effect, convert it into an NI-USRP RIO.<br />
|style="width: 10%; text-align: center;"|Tim Fountain<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-638<br />
|style="width: 30%;"| [[Running UHD and GNU Radio on NI-USRP RIO]]<br />
|style="width: 50%;"| This AN explains the process to updating your USRP-Rio to run UHD and GNU Radio. <br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-882<br />
|style="width: 30%;"| [[Synchronization and MIMO Capability with USRP Devices]]<br />
|style="width: 50%;"| Discusses the requirements for Multiple-In-Multiple-Out (MIMO) and phased-array systems. Summarizes the MIMO capability of each USRP device and daughterboard, and shows how to build MIMO systems with the USRP product family.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-309<br />
|style="width: 30%;"| [[About the Motherboard and Daughtercard EEPROM on USRP Devices]]<br />
|style="width: 50%;"| This AN discusses the EEPROM storage on various USRP devices and daughtercards. This guides explains how to update the EEPROM contents and recover from EEPROM corruption. The product codes, which are also stored in the EEPROM, for all USRP devices and daughtercards are also given for reference.<br />
|style="width: 10%; text-align: center;"|Trip Humphries<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-325<br />
|style="width: 30%;"| [[N200/N210 Device Recovery]]<br />
|style="width: 50%;"| This application note covers the details of recovering your N200/N210.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-504<br />
|style="width: 30%;"| [[USRP N Series Quick Start (Daughterboard Installation)]]<br />
|style="width: 50%;"| This application note is a detailed step-by-step guide to install a daughterboard into the USRP N200/N210.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-904<br />
|style="width: 30%;"| [[USRP X Series Quick Start (Daughterboard Installation)]]<br />
|style="width: 50%;"| This application note is a detailed step-by-step guide to install a daughterboard into the USRP X300/X310.<br />
|style="width: 10%; text-align: center;"|Neel Pandeya <br> Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-311<br />
|style="width: 30%;"| [[Software Development on the E310 and E312]]<br />
|style="width: 50%;"| This application note covers the software development process on the USRP E310 and E312. <br />
|style="width: 10%; text-align: center;"|Martin Braun<br>Nicolas Cuervo<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-296<br />
|style="width: 30%;"| [[Using Dual 10 Gigabit Ethernet on the USRP X300/X310]]<br />
|style="width: 50%;"| This short guide is meant to help in quickly setting up an X-series USRP for use over two 10 Gigabit Ethernet links simultaneously. <br />
|style="width: 10%; text-align: center;"|Paul David<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-823<br />
|style="width: 30%;"| [[Getting Started with RFNoC Development]]<br />
|style="width: 50%;"| This application note gives a brief introduction into the steps required to start developing RFNoC blocks on your computer. <br />
|style="width: 10%; text-align: center;"|Martin Braun<br>Nicolas Cuervo<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-178<br />
|style="width: 30%;"| [[Resolving Audio Codec Enumeration Issues On The E31x]]<br />
|style="width: 50%;"| This application note covers Resolving Audio Codec Enumeration Issues On The E31x. <br />
|style="width: 10%; text-align: center;"|Logan Fagg<br />
|-<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-244<br />
|style="width: 30%;"| [[Direction Finding with the USRP™ X-Series and TwinRX™]]<br />
|style="width: 50%;"| This application note covers using the USRP™ TwinRX™ daughterboard in a direction find application using the MUSIC algorithm. <br />
|style="width: 10%; text-align: center;"|Srikanth Pagadarai<br>Travis Collins<br>Alexander M. Wyglinski<br />
|-<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-335<br />
|style="width: 30%;"| [[Streaming processed data from the E31x with GNU Radio and ZMQ]]<br />
|style="width: 50%;"| This application note will demonstrate using the USRP E310 to remotely stream processed data to a host machine. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-121<br />
|style="width: 30%;"| [[Debugging FPGA images]]<br />
|style="width: 50%;"| This application note covers the basics to get you through the process of probing the signals inside an FPGA. In order to accomplish that, we will review briefly the 'Xilinx ChipScope Analyzer' and will apply it to one of our core RFNoC blocks: the RFNoC Signal generator. <br />
|style="width: 10%; text-align: center;"|Nicolas Cuervo<br>Sugandha Gupta <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-732<br />
|style="width: 30%;"| [[USRP E312 Battery Replacement Instructions]]<br />
|style="width: 50%;"| This application note covers replacing the battery cell inside the USRP E312. <br />
|style="width: 10%; text-align: center;"| Robin Coxe<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-305<br />
|style="width: 30%;"| [[X300/X310 Device Recovery]]<br />
|style="width: 50%;"| This application note covers the details of recovering the USRP X300/X310 via JTAG. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-832<br />
|style="width: 30%;"| [[Mapping Between ER-USRP and NI-USRP Product Numbers]]<br />
|style="width: 50%;"| This application note covers the details of the mapping between Ettus Research USRP and National Instruments USRP product numbers. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-315<br />
|style="width: 30%;"| [[Software Development on the E3xx USRP - Building RFNoC UHD / GNU Radio / gr-ettus from Source]]<br />
|style="width: 50%;"| This application note is one of a multi-part series which will cover the software development process on the USRP E310, E312 and E313. It will cover building the rfnoc-devel branch of UHD, GNU Radio and gr-ettus from source for the host machine, and cross-compiling the rfnoc-devel branch of UHD, GNU Radio and gr-ettus for the E3xx USRP. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-142<br />
|style="width: 30%;"| [[Transmitting DVB-S2 with GNU Radio and an USRP B210]]<br />
|style="width: 50%;"| This application note will demonstrate using an USRP B210 and the GNU Radio DTV example flowgraph to transmit a DVB-S2 video stream to an off-the-shelf satellite receiver. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-158<br />
|style="width: 30%;"| [[Using Ethernet-Based Synchronization on the USRP™ N3xx Devices]]<br />
|style="width: 50%;"| This application note provides instructions for synchronizing multiple USRP N3xx devices using White Rabbit Ethernet-based synchronization. <br />
|style="width: 10%; text-align: center;"| Dan Baker<br />
Wan Liu <br />
|-<br />
<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-642<br />
|style="width: 30%;"| [[Using the RFNoC Replay Block]]<br />
|style="width: 50%;"| This application note guides a user through basic use of the RFNoC Replay block and explains how to run the UHD Replay example. This example covers use on the X300/X310 and N310 products. <br />
|style="width: 10%; text-align: center;"| Wade Fife<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-630<br />
|style="width: 30%;"| [[Writing the USRP File System Disk Image to a SD Card]]<br />
|style="width: 50%;"| This application note will provide step-by-step instructions on writing a file system disk image to a SD card using Linux. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-524<br />
|style="width: 30%;"| [[Building and Installing UHD and GNU Radio in an Offline Environment]]<br />
|style="width: 50%;"| This application note will provide step-by-step instructions on building and installing UHD and GNU Radio in an offline environment. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-525<br />
|style="width: 30%;"| [[Building and Installing UHD and GNU Radio to a Custom Prefix]]<br />
|style="width: 50%;"| This application note provides step-by-step instructions on building and installing UHD and GNU Radio to a local directory. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-725<br />
|style="width: 30%;"| [[USRP N320/N321 LO Distribution]]<br />
|style="width: 50%;"| This application note provides an overview of using the LO Distribution of the N320/N321 USRPs.<br />
<br />
|style="width: 10%; text-align: center;"| Brian Avenell <br />
|-<br />
<br />
<br />
|style="width: 10%; text-align: center;"| AN-452<br />
|style="width: 30%;"| [[5G NR EVM Measurements with the USRP N320/N321]]<br />
|style="width: 50%;"| Example EVM measurements are shown using the USRP N320/N321 receiver and the 5G New Radio (5G NR) modulation standard. The use of I/Q image calibration and spur-dodging are demonstrated as methods to improve EVM performance. <br />
|style="width: 10%; text-align: center;"| Drew Fischer<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-620<br />
|style="width: 30%;"| [[Troubleshooting X300/X310 Device Discovery Issues]]<br />
|style="width: 50%;"| Troubleshooting guide to intended to cover some of the most commonly recommended steps to enable USRP connectivity. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-088<br />
|style="width: 30%;"| [[USRP Host Performance Tuning Tips and Tricks]]<br />
|style="width: 50%;"| This application note provides various tips and tricks for tuning your host computer for best performance when working with USRP devices. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-355<br />
|style="width: 30%;"| [[Modifying an X310 Chassis for External LO Sharing]]<br />
|style="width: 50%;"| This document describes how to modify an X310 chassis to wire the LO out of the back plate. Doing this will allow the user to export and import an LO signal as desired when using a compatible daughterboard such as the TwinRX. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-500<br />
|style="width: 30%;"| [[Getting Started with DPDK and UHD]]<br />
|style="width: 50%;"| This application note walks through the process to get started with the Data Plane Development Kit (DPDK) driver within UHD. <br />
|style="width: 10%; text-align: center;"| Nate Temple<br />
Alex Williams<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-800<br />
|style="width: 30%;"| [[Enabling Ethernet Connectivity on Octoclock and Octoclock-G]]<br />
|style="width: 50%;"| This document supplements the UHD Manual's guide for updating the Octoclock bootloader to allow for Ethernet communications with the device. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-621<br />
|style="width: 30%;"| [[Troubleshooting N310/N320 Device Discovery Issues]]<br />
|style="width: 50%;"| Troubleshooting guide to intended to cover some of the most commonly recommended steps to enable USRP connectivity. Serves as a supplement to the N3xx getting started guide. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
<br />
|style="width: 10%; text-align: center;"| AN-883<br />
|style="width: 30%;"| [[Synchronizing USRP Events Using Timed Commands in UHD]]<br />
|style="width: 50%;"| Guide to cover common USRP synchronization scenarios and deep-dive into the use of timed commands within USRPs. <br />
|style="width: 10%; text-align: center;"| Sam Reiter<br />
|-<br />
|}</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4878Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T03:45:43Z<p>SamReiter: /* Application Note Number */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-883'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-03-1 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; // only the bottom 4 lines: 0xF = 00001111 = Pin 0, 1, 2, 3<br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.00<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps(); // define t=0<br />
<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // reset HIGH (async)<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=2<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // set HIGH @ t=4<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=6<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4877Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T03:44:25Z<p>SamReiter: /* Revision History */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-03-1 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; // only the bottom 4 lines: 0xF = 00001111 = Pin 0, 1, 2, 3<br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.00<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps(); // define t=0<br />
<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // reset HIGH (async)<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=2<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // set HIGH @ t=4<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=6<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4876Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T03:41:16Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; // only the bottom 4 lines: 0xF = 00001111 = Pin 0, 1, 2, 3<br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.00<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps(); // define t=0<br />
<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // reset HIGH (async)<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=2<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // set HIGH @ t=4<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=6<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4875Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T03:39:10Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; // only the bottom 4 lines: 0xF = 00001111 = Pin 0, 1, 2, 3<br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.0s<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps();<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // reset HIGH (async)<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=2<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0); // set HIGH @ t=4<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0); // set LOW @ t=6<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4874Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T02:40:41Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; // only the bottom 4 lines: 0xF = 00001111 = Pin 0, 1, 2, 3<br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.0s<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps();<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4873Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T02:40:31Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; // only the bottom 4 lines: 0xF = 00001111 = Pin 0, 1, 2, $ 3<br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.0s<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps();<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4872Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T02:36:42Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; <br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.0s<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps();<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
[[File:gpio example.png|1000px|center]]<br />
<center>Figure 4 - Logic analyzer readout from timed GPIO example</center><br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4871Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T02:35:35Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; <br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.0s<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps();<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
File:gpio example.png<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=File:gpio_example.png&diff=4870File:gpio example.png2020-03-02T02:35:13Z<p>SamReiter: </p>
<hr />
<div></div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4869Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T02:25:47Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low. This can be copy-pasted into existing USRP examples:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; <br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.0s<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps();<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4868Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T02:20:12Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
The following code uses timed commands to vary a USRP's bottom 4 GPIO lines between high and low:<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; <br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.0s<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps();<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
A low-cost logic analyzer can be used to monitor the GPIO line states:<br />
<br />
<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4867Synchronizing USRP Events Using Timed Commands in UHD2020-03-02T02:17:34Z<p>SamReiter: /* Example: Using Timed Commands to Control GPIO */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
<pre><br />
/********************************************/<br />
/*********** begin gpio operations **********/<br />
/********************************************/<br />
<br />
// set up some catch-all masks<br />
uint32_t gpio_line = 0xF; <br />
uint32_t all_one = 0xFF;<br />
uint32_t all_zero = 0x00;<br />
<br />
// reset usrp time to 0.0s<br />
usrp->set_time_source("internal");<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(2000));<br />
<br />
uhd::time_spec_t now_time = usrp->get_time_last_pps();<br />
<br />
// set gpio pins up for output<br />
usrp->set_gpio_attr("FP0", "DDR", all_one, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "CTRL", all_zero, gpio_line, 0);<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(2.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
// set all gpio lines to output 1<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(4.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_one, gpio_line, 0);<br />
<br />
// set all gpio lines to output 0<br />
usrp->clear_command_time();<br />
usrp->set_command_time(now_time + uhd::time_spec_t(6.0));<br />
usrp->set_gpio_attr("FP0", "OUT", all_zero, gpio_line, 0);<br />
<br />
usrp->clear_command_time();<br />
<br />
/********************************************/<br />
/*********** quit gpio operations ***********/<br />
/********************************************/<br />
</pre><br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4866Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T17:37:45Z<p>SamReiter: /* General IP Core Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that fall into this category. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
IP Cores like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4865Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T17:36:57Z<p>SamReiter: /* General RFNoC Block Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General IP Core Timing===<br />
The next case to cover is the handling of timed commands within FPGA blocks that are not the Radio Core. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default USRP FPGA blocks that would be considered IP Cores. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4864Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T17:35:09Z<p>SamReiter: /* Command Queue */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every IP core on a USRP, including the Radio Core, DDC, DUC, and custom blocks, includes one command queue per data stream (certain blocks are designed to pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4863Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T17:33:33Z<p>SamReiter: /* Command Queue */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in the USRP do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4862Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T17:33:02Z<p>SamReiter: /* Command Queue */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other IP cores (including custom RFNoC blocks) and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4861Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T17:31:03Z<p>SamReiter: /* Command Queue */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (IP Cores)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4860Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T17:22:23Z<p>SamReiter: /* Command Queue */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|8<br />
|5<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|8<br />
|5<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|64<br />
|64<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|64<br />
|64<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4859Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T17:07:09Z<p>SamReiter: /* Command Queue */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Xilinx Kintex-7 XC7K325T<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Xilinx Kintex-7 XC7K410T<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Xilinx Zynq 7020 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Xilinx Zynq 7045 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N300<br />
|Xilinx Zynq-7035 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|N310/N320/N321<br />
|Xilinx Zynq-7100 SoC<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Xilinx Spartan 6 XC6SLX75<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Xilinx Spartan 6 XC6SLX150<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Xilinx Spartan 3A-DSP XC3SD1800A<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Xilinx Spartan 3A-DSP XC3SD3400A<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4858Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T16:48:08Z<p>SamReiter: /* Command Queue */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|8<br />
|5<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|8<br />
|5<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|8<br />
|5<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|8<br />
|5<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|8<br />
|5<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4857Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T16:32:19Z<p>SamReiter: /* Command Queue */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure 3). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4856Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T16:31:48Z<p>SamReiter: /* Radio Core Block Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<center>Figure 3 - RFNoC Architecture with UBX daughterboard</center><br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4855Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T16:30:44Z<p>SamReiter: /* CHDR Packet Types and Structure */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<center>Figure 2 - Expanded view of CHDR packet composition</center><br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4854Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T16:30:05Z<p>SamReiter: /* A System-Level Example */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<center>Figure 1 - Example system configuration with Host PC, Octoclock, and X310 radios</center><br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4853Synchronizing USRP Events Using Timed Commands in UHD2020-02-26T16:28:16Z<p>SamReiter: </p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4848Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:56:24Z<p>SamReiter: /* USRP Stream Cmd (RX timing) */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd ([https://files.ettus.com/manual/structuhd_1_1rx__metadata__t.html rx_metadata_t Struct Reference])====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4847Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:54:55Z<p>SamReiter: /* UHD API for Event Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd (RX timing)====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4846Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:54:21Z<p>SamReiter: /* Timed Commands */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd (RX timing)====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
====Timed Commands====<br />
<br />
Clear the command time so future commands are sent ASAP.<br />
<br />
clear_command_time()<br />
<br />
Set the time at which the control commands will take effect.<br />
<br />
set_command_time()<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4845Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:52:25Z<p>SamReiter: /* USRP Stream Cmd (RX timing) */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd (RX timing)====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd(stream_cmd)<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4844Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:51:45Z<p>SamReiter: /* USRP Stream Cmd (RX timing) */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd (RX timing)====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
Issue a stream command to the USRP:<br />
<br />
issue_stream_cmd()<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4843Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:49:39Z<p>SamReiter: /* Overview of Relevant Methods */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd (RX timing)====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
Member function indicating whether the stream will begin immediately:<br />
<br />
stream_cmd.stream_now<br />
<br />
Indicates the point in time for an RX stream to begin:<br />
<br />
stream_cmd.time_spec<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4842Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:47:07Z<p>SamReiter: /* UHD API for Event Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Cmd (RX timing)====<br />
<br />
Default stream command constructor:<br />
<br />
stream_cmd_t()<br />
<br />
<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4841Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:43:17Z<p>SamReiter: /* UHD API for Event Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP TX Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4840Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:40:36Z<p>SamReiter: /* USRP Metadata (TX timing) */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP Metadata ([https://files.ettus.com/manual/structuhd_1_1tx__metadata__t.html tx_metadata_t Struct Reference])====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4839Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:38:51Z<p>SamReiter: /* USRP Metadata (TX timing) */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
Default TX metadata constructor: <br />
<br />
tx_metadata_t()<br />
<br />
Indicates whether a TX packet has a timestamp included:<br />
<br />
md.has_time_spec<br />
<br />
Sets the timestamp that will be added to a TX packet<br />
<br />
md.time_spec<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4838Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:31:21Z<p>SamReiter: /* Get USRP Time */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4837Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:31:08Z<p>SamReiter: /* UHD API for Event Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some less common methods which require a more detailed explanation can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get a list of possible time sources:<br />
<br />
get_time_sources()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4836Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:30:16Z<p>SamReiter: /* UHD API for Event Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some methods require a more detailed explanation in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get a list of possible time sources:<br />
<br />
get_time_sources()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Synchronize the times across all motherboards in this configuration:<br />
<br />
set_time_unknown_pps()<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4835Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:29:38Z<p>SamReiter: /* UHD API for Event Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. Some methods require a more detailed explanation in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html The multi_usrp Class Reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get a list of possible time sources:<br />
<br />
get_time_sources()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
Sets the time registers on the USRP immediately:<br />
<br />
set_time_now()<br />
<br />
The above call is not safe for multi-device sync when all USRPs are Octoclock slaves.<br />
<br />
Set the time registers on the usrp at the next pps tick:<br />
<br />
set_time_next_pps()<br />
<br />
Synchronize the times across all motherboards in this configuration:<br />
<br />
set_time_unknown_pps()<br />
<br />
See manual for details on the use of <code>set_time_unknown_pps()</code>.<br />
<br />
Set the time source for the USRP device:<br />
<br />
set_time_source()<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4834Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:20:59Z<p>SamReiter: /* UHD API for Event Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. More detail can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html the multi_usrp class reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get a list of possible time sources:<br />
<br />
get_time_sources()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====Set USRP Time====<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4833Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:20:42Z<p>SamReiter: /* Get and Set USRP Time */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. More detail can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html the multi_usrp class reference].<br />
<br />
====Get USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get a list of possible time sources:<br />
<br />
get_time_sources()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4832Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:20:04Z<p>SamReiter: /* Get and Set USRP Time */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. More detail can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html the multi_usrp class reference].<br />
<br />
====Get and Set USRP Time====<br />
<br />
Get the current time in the USRP time registers:<br />
<br />
get_time_now()<br />
<br />
Get the time when the last pps pulse occurred:<br />
<br />
get_time_last_pps()<br />
<br />
Get a list of possible time sources:<br />
<br />
get_time_sources()<br />
<br />
Get the currently set time source:<br />
<br />
get_time_source()<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4831Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:14:30Z<p>SamReiter: /* UHD API for Event Timing */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
In this section, we will list and briefly describe commands that are commonly used in USRP event timing. More detail can be found in [https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html the multi_usrp class reference].<br />
<br />
====Get and Set USRP Time====<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4830Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:11:09Z<p>SamReiter: /* UHD API for Timed Commands */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Event Timing==<br />
<br />
====Get and Set USRP Time====<br />
<br />
====USRP Metadata (TX timing)====<br />
<br />
====USRP Stream Args (RX timing)====<br />
<br />
====Timed Commands====<br />
<br />
===Overview of Relevant Methods===<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiterhttps://kb.ettus.com/index.php?title=Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD&diff=4829Synchronizing USRP Events Using Timed Commands in UHD2020-02-18T00:08:07Z<p>SamReiter: /* UHD API for Timed Commands */</p>
<hr />
<div>==Application Note Number==<br />
'''AN-TC'''<br />
<br />
==Revision History==<br />
{| class="wikitable"<br />
!Date<br />
!Author<br />
!Details<br />
|-<br />
|style="text-align:center;"| 2020-02-10 <br />
|style="text-align:center;"| Sam Reiter<br />
|style="text-align:center;"| Initial creation<br />
|}<br />
<br />
==Abstract==<br />
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.<br />
<br />
==Timed Commands: Overview and Usecases==<br />
<br />
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command use cases include:<br />
<br />
<ul><br />
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.<br />
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]). <br />
<li> Changing the state of a USRP's GPIO line at an absolute time. <br />
<li> Coordinating a simultaneous change of gain and frequency in a USRP.<br />
<li> Scheduling frequency hopping at set time increments.<br />
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts. <br />
</ul><br />
<br />
<br />
===A System-Level Example===<br />
A common use case of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the low level details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.<br />
<br />
There are four key elements required for phase coherent operation of resync-capable USRPs:<br />
<ol><br />
<li> All USRPs share a common reference clock (10MHz Ref)<br />
<li> All USRPs share a common sense of time (PPS)<br />
<li> LO and DSP tuning is synchronous<br />
<li> Streaming is started synchronously <br />
</ol><br />
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:<br />
<br />
[[File:Mimo Setup.png|1200px|center]]<br />
<br />
<br />
Now that the system has all the physical connections necessary, we need to coordinate things in our host code. This will include importing the 10MHz Ref and PPS as well as ensuring synchronous tuning and streaming. <br />
<br />
First, we need to create a <code>multi_usrp</code> object that contains both USRPs in our physical configuration:<br />
<br />
uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make("addr0=192.168.10.2,addr1=192.168.10.3");<br />
<br />
Next, we'll let the USRPs know that they should expect a 10MHz clock on their "Ref In" port, and a PPS signal on their "PPS In" port:<br />
<br />
usrp->set_clock_source("external"); <br />
usrp->set_time_source("external");<br />
<br />
At this point, both USRPs are locked to an external reference and a common PPS signal. Next we want to tell both USRPs to reset their sense of time to 0.000s on the next PPS edge:<br />
<br />
usrp->set_time_next_pps(uhd::time_spec_t(0.0));<br />
std::this_thread::sleep_for(std::chrono::milliseconds(1000));<br />
<br />
We won't cover it much in this application note, but for now understand <code>time_spec_t</code> is the UHD API's means of formatting time values for a USRP. After we reset the USRP's sense of time, we wait 1 second to ensure a PPS rising edge occurs and latches the 0.000s value to both USRPs. At this point, both USRPs should have a shared sense of time. We've now satisfied the first and second requirements for phase coherent USRP operation. Let's move on to synchronously tuning using timed commands:<br />
<br />
usrp->clear_command_time();<br />
<br />
usrp->set_command_time(usrp->get_time_now() + uhd::time_spec_t(0.1)); //set cmd time for .1s in the future<br />
<br />
uhd::tune_request_t tune_request(freq);<br />
usrp->set_rx_freq(tune_request);<br />
std::this_thread::sleep_for(std::chrono::milliseconds(110)); //sleep 110ms (~10ms after retune occurs) to allow LO to lock<br />
<br />
usrp->clear_command_time();<br />
<br />
With the above code block, we are able to set a command time equal to the current time + 1s. Any commands that are called after <code>set_command_time()</code> will be sent to the USRP with a timestamp corresponding to the argument passed to <code>set_command_time()</code>. Because of this timestamp, the USRP will wait until the command time passes to execute the tune request. This will ensure that the LO and DSP chain of our USRPs are retuned synchronously (on the same clock cycle), satisfying the third requirement for phase coherent operation.<br />
<br />
The final step in configuring our system is to set up synchronous streaming from both radios. For this example, we'll set up synchronous RX streaming using the following code:<br />
<br />
// create a receive streamer<br />
uhd::stream_args_t stream_args("fc32", wire); // complex floats<br />
stream_args.channels = "0,1,2,3";<br />
uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args);<br />
<br />
// setup streaming<br />
uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS);<br />
stream_cmd.stream_now = false;<br />
stream_cmd.time_spec = uhd::time_spec_t(usrp->get_time_now() + uhd::time_spec_t(1.0));<br />
rx_stream->issue_stream_cmd(stream_cmd);<br />
<br />
Our system will begin to stream data 1s after the time returned by <code>usrp->get_time_now()</code>. Data sent to the host will be phase coherent between the 4 RX channels in this system. This architecture allows a system with X310 + UBX daughterboards to maintain a known phase relationship between channels across runs of code and system power cycles.<br />
<br />
==Clocking and Timekeeping in the USRP==<br />
<br />
In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD. This is the low level detail section of this application note.<br />
<br />
===PPS (Pulse Per Second)===<br />
<br />
PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:<br />
<br />
*1Hz<br />
*TTL Signal Levels<br />
*25% duty cycle<br />
<br />
A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).<br />
<br />
A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the <code>vita_time</code> counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as: <br />
<br />
set_time_next_pps<br />
<br />
set_time_unknown_pps<br />
<br />
Ettus Research recommends the [https://www.ettus.com/all-products/octoclock/ Octoclock] for distribution of PPS and 10MHz REF signals across multiple devices, and the [https://www.ettus.com/all-products/octoclock-g/ Octoclock-G] for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the [https://files.ettus.com/manual/page_sync.html UHD Manual: Device Synchronization].<br />
<br />
===CHDR Packet Types and Structure===<br />
<br />
CHDR (pronounced "Cheddar", like the cheese) or "Compressed Header" packets are the packet type used to pass data and commands between a host computer and USRP. CHDR is a derivative of the VITA 49 (VRT) protocol which is proprietary to USRP devices. See [https://files.ettus.com/manual/page_rtp.html UHD Manual: Radio Transport Protocols] for more information.<br />
<br />
There are 4 types of packets used in the USRP:<br />
*Data <br />
*Flow Control<br />
*Command <br />
*Command Response<br />
<br />
The type of CHDR packet is determined by the value of bits 63 and 62 in the CHDR header: <br />
<br />
[[File:packet breakdown.png|1200px|center]]<br />
<br />
<br />
<br />
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.<br />
<br />
===Command Queue===<br />
As commands are passed from host to USRP, they are added to a FIFO on the USRP's FPGA. This FIFO is called the command queue and all commands that are sent to the USRP must pass through a command queue. Each block in the FPGA that handles data also has its own command queue. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**). <br />
<br />
Every command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the command queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on-the-fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed. <br />
<br />
Every RFNoC block, including the Radio Core, includes one command queue per data stream (certain blocks can pass multiple data streams). The depth of this command queue varies from device to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA. Here are the default command queue depths for various USRP models:<br />
<br />
<br />
{| class="wikitable" style="margin:auto;"<br />
!USRP<br />
!FPGA<br />
!Default CMD Queue Depth (Radio Core)<br />
!Default CMD Queue Depth (RFNoC Blocks)<br />
|-<br />
<br />
|X300<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|X310<br />
|Kintex 7<br />
|0<br />
|0<br />
|-<br />
<br />
|E310/E312/E313<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|E320<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|N300/N310/N320/N321<br />
|Zynq<br />
|0<br />
|0<br />
|-<br />
<br />
|B200/B200mini<br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|B210/B205mini <br />
|Spartan 6<br />
|0<br />
|0<br />
|-<br />
<br />
|N200<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|N210<br />
|Spartan<br />
|0<br />
|0<br />
|-<br />
<br />
|}<br />
<center>Table 1 - Default command queue depth of various USRPs</center><br />
<br />
===Radio Core Block Timing===<br />
<br />
The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signal path, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling. <br />
<br />
[[File:rfnoc with ubx.png|1200px|center]]<br />
<br />
In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or <code>vita_time</code> in UHD. The <code>vita_time</code> counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and sample rate) of packets which pass through these blocks.<br />
<br />
===General RFNoC Block Timing===<br />
The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts on IQ samples that are passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.<br />
<br />
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the sample rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included timestamp equal to <code>vita_time</code>) and are then passed downstream to the DDC. The DDC will read the timestamps of the incoming samples and apply any queued frequency shift precisely on the first sample with a timestamp >= the command time. <br />
<br />
With this method, a USRP can begin receiving data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.<br />
<br />
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included in the metadata. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.<br />
<br />
===Miscellaneous Timed Command Notes===<br />
<br />
====Types of timed commands====<br />
* The majority timing operations amount a sequence of peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc. The timestamp of a timed command corresponds to the beginning of this peek/poke sequence. <br />
* The timing of data transmission (TX) is a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission. This is handled by the FPGA when a timestamp is present in the first TX sample's metadata.<br />
* A timed receive operation (RX) requires that a stream command with a timestamp be issued to the radio (see <code>stream_cmd.time_spec</code> in the "System Level Example"). This syntax is different than a TX operation and does not involve sample metadata.<br />
<br />
====Timed commands on AD93xx-based devices====<br />
*Timed commands are supported on AD93xx-based USRPs (E3xx, B2xx, N300/N310), with a few caveats:<br />
**Timed commands do not allow for phase resync of the AD93xx internal LO.<br />
**All functions that directly interact with the AD93xx (tuning, gain change, etc) are subject to the scheduling of the AD93xx. The determinism of these operations are not guaranteed.<br />
*Timed commands for TX and RX on AD936x devices are supported, with caveats:<br />
**There will be a delay between an absolute time passing and the AD936x actually beginning a streaming operation. This delay has been observed to be consistent ''for streaming'', but other operations like gain setting require SPI readback from the RFIC and are non-deterministic.<br />
**This is to say that a timed streaming command to begin TX and RX at the same time, on an AD936x-based device (which is in external loopback) should result in a consistent delay between TX and RX down to the sample. Other RFIC operations like changing gain or tuning LO frequency will not have this consistency.<br />
<br />
====Are timed commands blocking?====<br />
*No. Timed commands are passed from host to USRP and are added to a queue while host code progresses.<br />
<br />
===Summary===<br />
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.<br />
<br />
==UHD API for Timed Commands==<br />
<br />
===Overview of Relevant Methods===<br />
<br />
===Commands that can be timed===<br />
<br />
==Example: Using Timed Commands to Phase Align Channels==<br />
<br />
==Example: Using Timed Commands to Control GPIO==<br />
<br />
[[Category:Application Notes]]</div>SamReiter