Difference between revisions of "Synchronizing USRP Events Using Timed Commands in UHD"

From Ettus Knowledge Base
Jump to: navigation, search
(Command Queue)
(A System-Level Example)
(38 intermediate revisions by the same user not shown)
Line 14: Line 14:
  
 
==Abstract==
 
==Abstract==
This AN discusses Timed Commands in UHD. We will explore usecases, theory of operation, and examples of timed command use in UHD 3.x.
+
This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.
  
 
==Timed Commands: Overview and Usecases==
 
==Timed Commands: Overview and Usecases==
*A  
+
Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command usecases include:
*B
+
<ul>
*C
+
<li> Configuring multiple channels of a single USRP to change frequency simultaneously.
*D
+
<li> Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels ([https://kb.ettus.com/Synchronization_and_MIMO_Capability_with_USRP_Devices#MIMO_Capability_of_Ettus_Research_USRP_Devices if supported by daughterboard]).
*E
+
<li> Changing the state of a USRP's GPIO line at an absolute time.
*F
+
<li> Coordinating a simultaneous change of gain and frequency in a USRP.
*G
+
<li> Scheduling frequency hopping at set time increments.
*H
+
<li> Using RFNoC blocks like the Replay Block to transmit phase coherent bursts.
*I
+
</ul>
*J
+
 
 +
===A System-Level Example===
 +
A frequent usecase of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.
 +
 
 +
There are four key elements required for phase coherent operation of resync-capable USRPs:
 +
<ol>
 +
<li> All USRPs share a common reference clock (10MHz Ref)
 +
<li> All USRPs share a common sense of time (PPS)
 +
<li> LO and DSP tuning is synchronous
 +
<li> Streaming is started synchronously
 +
</ol>
 +
LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:
 +
 
 +
[[File:packet breakdown.png|1400px|center]]
  
 
==Clocking and Timekeeping in the USRP==
 
==Clocking and Timekeeping in the USRP==
Line 48: Line 61:
  
  
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “Timed Commands”.
+
All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.
  
 
===Command Queue===
 
===Command Queue===
 
The command queue is a FIFO for all commands that are sent to an RFNoC block. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**).  
 
The command queue is a FIFO for all commands that are sent to an RFNoC block. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**).  
  
Every RFNoC command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this Application Note. When commands enter the command queue, their timestamp is compared against the queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on the fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed.   
+
Every RFNoC command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on the fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed.   
  
Every RFNoC block, including the Radio Core, includes 1 command queue per port (there's generally 1 port per data stream). The depth of this command queue varies from device-to device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA.
+
Every RFNoC block, including the Radio Core, includes one command queue per port (there's generally one port per data stream). The depth of this command queue varies from device-to-device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA.
 +
 
 +
 
 +
{| class="wikitable" style="margin:auto;"
 +
!USRP
 +
!FPGA
 +
!Default CMD Queue Depth (Radio Core)
 +
!Default CMD Queue Depth (RFNoC Blocks)
 +
|-
 +
 
 +
|X300
 +
|Kintex 7
 +
|0
 +
|0
 +
|-
 +
 
 +
|X310
 +
|Kintex 7
 +
|0
 +
|0
 +
|-
 +
 
 +
|E310/E312/E313
 +
|Zynq
 +
|0
 +
|0
 +
|-
 +
 
 +
|E320
 +
|Zynq
 +
|0
 +
|0
 +
|-
 +
 
 +
|N300/N310/N320/N321
 +
|Zynq
 +
|0
 +
|0
 +
|-
 +
 
 +
|B200/B200mini
 +
|Spartan 6
 +
|0
 +
|0
 +
|-
 +
 
 +
|B210/B205mini
 +
|Spartan 6
 +
|0
 +
|0
 +
|-
 +
 
 +
|N200
 +
|Spartan
 +
|0
 +
|0
 +
|-
 +
 
 +
|N210
 +
|Spartan
 +
|0
 +
|0
 +
|-
 +
 
 +
|}
 +
<center>Table 1 - Default command queue depth of various USRPs</center>
  
 
===PPS (Pulse Per Second)===
 
===PPS (Pulse Per Second)===
Line 88: Line 166:
 
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the IQ rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included absolute timestamp) and are then passed downstream to the DDC. The DDC will read the timestamp of the incoming CHDR packet, extract the IQ samples, and apply any queued frequency shift at the specified time in the stream. To get sample-precise data manipulation, the DDC can interpolate timestamps for each IQ sample by incrementing its sense of time by <code>1/samp_rate</code> for each IQ sample.  
 
RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the IQ rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included absolute timestamp) and are then passed downstream to the DDC. The DDC will read the timestamp of the incoming CHDR packet, extract the IQ samples, and apply any queued frequency shift at the specified time in the stream. To get sample-precise data manipulation, the DDC can interpolate timestamps for each IQ sample by incrementing its sense of time by <code>1/samp_rate</code> for each IQ sample.  
  
With this method, a USRP can begin RX'ing data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC at the point in data corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.
+
With this method, a USRP can begin RX'ing data at absolute time <code>t0</code>, tune it's LO at an absolute time of <code>t1</code>, and then apply a frequency shift in the DDC <u>at the point in data</u> corresponding to <code>t1</code>, resulting in a physical and digital frequency change that begin on the exact same sample.
  
 
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.
 
The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from <code>ctrl_iface.cpp</code>.
Line 99: Line 177:
  
 
====Timed commands on AD93xx-based devices====
 
====Timed commands on AD93xx-based devices====
*Support for timed commands on AD93xx devices has been added, but it does not allow for phase resync of the internal LO.
+
*Support for timed commands on AD93xx devices has been added.
*Timed commands for TX and RX on AD936x devices are supported, with caveats
+
**It does not allow for phase resync of the AD93xx internal LO.
 +
**All non-Radio Core blocks all operate as you'd expect on USRPs with a dedicated daughterboard.
 +
*Timed commands for TX and RX on AD936x devices are supported, with caveats:
 
**There will be a delay between an absolute time passing and the AD936x actually beginning an operation (TX/RX/Gain/Tune). This delay is deterministic if characterized ''for a given command''.
 
**There will be a delay between an absolute time passing and the AD936x actually beginning an operation (TX/RX/Gain/Tune). This delay is deterministic if characterized ''for a given command''.
 
**SPI communication between the FPGA and RFIC means that the delay between an absolute time passing varies depending on the timed command issued.  
 
**SPI communication between the FPGA and RFIC means that the delay between an absolute time passing varies depending on the timed command issued.  
Line 106: Line 186:
  
 
===Summary===
 
===Summary===
 
+
Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the <code>vita_time</code>. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.
==USRP Time Sources==
+
 
+
===Internal PPS===
+
 
+
===External PPS===
+
 
+
===GPSDO===
+
  
 
==UHD API for Timed Commands==
 
==UHD API for Timed Commands==
  
 
===Overview of Relevant Methods===
 
===Overview of Relevant Methods===
 
===Timed Commands vs. Stream Commands with time_spec (remove this?)===
 
  
 
===Commands that can be timed===
 
===Commands that can be timed===

Revision as of 11:18, 14 February 2020

Application Note Number

AN-TC

Revision History

Date Author Details
2020-02-10 Sam Reiter Initial creation

Abstract

This AN discusses Timed Commands in UHD. We will explore use cases, theory of operation, and multi_usrp based examples of timed command used in UHD 3.x.

Timed Commands: Overview and Usecases

Timed Commands are an important aspect of using a USRP. They allow a user to coordinate USRP state changes with nanosecond precision across multiple devices. Examples of timed command usecases include:

  • Configuring multiple channels of a single USRP to change frequency simultaneously.
  • Configuring the channels on multiple USRPs to synchronously retune and ensure a predictable phase offset between channels (if supported by daughterboard).
  • Changing the state of a USRP's GPIO line at an absolute time.
  • Coordinating a simultaneous change of gain and frequency in a USRP.
  • Scheduling frequency hopping at set time increments.
  • Using RFNoC blocks like the Replay Block to transmit phase coherent bursts.

A System-Level Example

A frequent usecase of timed commands is to ensure a predictable and repeatable phase offset between the channels of multiple USRPs. Before we dive into the details of timed commands, lets take a high level look at where they are used in configuring a phase coherent MIMO system.

There are four key elements required for phase coherent operation of resync-capable USRPs:

  1. All USRPs share a common reference clock (10MHz Ref)
  2. All USRPs share a common sense of time (PPS)
  3. LO and DSP tuning is synchronous
  4. Streaming is started synchronously

LO sharing is implemented in some USRP products, but will not be covered in this example. Here is a physical configuration satisfying the requirements listed above:

packet breakdown.png

Clocking and Timekeeping in the USRP

In this section, we will cover several key topics relating to USRP synchronization and the use of timed commands in UHD.

CHDR Packet Types and Structure

CHDR or "Compressed Header" packets are a derivative of the VITA 49 (VRT) protocol. While the CHDR protocol is proprietary to USRP devices, mapping to and from VITA-49 is a trivial task. All packet routing in b2xx and "Gen 3" USRPs uses the CHDR protocol.

There are 4 types of packets used in the USRP:

  • Data
  • Flow Control
  • Command
  • Command Response

The type of packet is determined by the state of bits 63 and 62 in the CHDR header. See UHD Manual: Radio Transport Protocols for more information on CHDR structure.

packet breakdown.png


All of these packet types have a single bit (61) used to denote whether an optional timestamp is included. If present, a timestamp is a 64-bit value representing an absolute time value. In this application note, we’re concerned with the use and functionality of command packets with a timestamp present, also known as “timed commands”.

Command Queue

The command queue is a FIFO for all commands that are sent to an RFNoC block. The command queue FIFO is not to be confused with the data FIFOs used to buffer data between blocks (pictured in Figure **###**).

Every RFNoC command queue maintains a sense of time. The mechanism for acquiring this sense of time is different between the Radio Core and other RFNoC blocks and will be explored later in this application note. When commands enter the command queue, their timestamp is compared against the queue's sense of time and the commands are executed when Queue Time >= Command Time. Commands without timestamps are executed immediately when they're at the front of the queue. Command queues in RFNoC do not support on the fly reordering, meaning a command at the front of the queue will block subsequent commands from executing even if their timestamp has passed.

Every RFNoC block, including the Radio Core, includes one command queue per port (there's generally one port per data stream). The depth of this command queue varies from device-to-device is determined at FPGA compilation time based on user settings and available resources. An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA.


USRP FPGA Default CMD Queue Depth (Radio Core) Default CMD Queue Depth (RFNoC Blocks)
X300 Kintex 7 0 0
X310 Kintex 7 0 0
E310/E312/E313 Zynq 0 0
E320 Zynq 0 0
N300/N310/N320/N321 Zynq 0 0
B200/B200mini Spartan 6 0 0
B210/B205mini Spartan 6 0 0
N200 Spartan 0 0
N210 Spartan 0 0
Table 1 - Default command queue depth of various USRPs

PPS (Pulse Per Second)

PPS is a signal used by USRPs for time synchronization. With a shared PPS, the sense of time can be aligned across several USRPs, allowing for the synchronization of timed command execution on an arbitrary number of radio channels. Within the context of a USRP, a PPS signal is expected to have the following properties:

  • 1Hz
  • TTL Signal Levels
  • 25% duity cycle

A USRP's PPS can be derived from a GPSDO automatically, from an externally supplied PPS signal, or via internal PPS synthesis (not supported in legacy USRPs).

A PPS trigger is used to coordinate time alignment events across multiple devices. For example, the USRPs internal sense of time (the vita_time counter) can be synchronously set/reset across multiple USRPs via UHD API calls such as

set_time_next_pps
set_time_unknown_pps

Ettus Research recommends the Octoclock for distribution of PPS and 10MHz REF signals across multiple devices, and the Octoclock-G for this functionality as well as tight alignment with GPS time. For further information on PPS and other common reference signals, see the UHD Manual: Device Synchronization.

Radio Core Block Timing

The Radio Core is the heart of the USRP's functionality. The radio core is responsible for controlling all TX and RX daughterboard components (synthesizer, signalpath, gain and attenuation elements, etc.), GPIO, setting up data streaming to/from DACs and ADCs, and related error handling.

rfnoc with ubx.png

In addition to the functionality listed above, the Radio Core is also responsible for maintaining the USRP's sense of absolute time. This absolute time value is stored as a 64-bit counter called the "VITA time" or vita_time in UHD. The vita_time counter is incremented off of the FPGA's base clock and local to the Radio Core USRP, meaning that other RFNoC blocks can not reference this counter value directly. This sense of absolute time is a subtle yet important difference between the Radio Core and other RFNoC blocks, which can also execute timed commands but do so based on the timestamps (and IQ rate) of packets which pass through these blocks.

General RFNoC Block Timing

The next case to cover is the handling of timed commands within a "General" RFNoC block. The Digital Down Converter (DDC) and Digital Up Converter (DUC) are two default RFNoC blocks that would be considered general RFNoC blocks. These blocks are responsible for performing digital frequency shifts of IQ data that is passed through them. Precise execution of these frequency shifts are essential to phase coherent operation of the USRP.

RFNoC blocks like the DDC and DUC are reliant on the timestamp of packets (and the IQ rate of the radio) to derive a sense of time. In the case of the receive chain, samples that are digitized by an ADC are then packaged into a CHDR Packet by the Radio Core (with an included absolute timestamp) and are then passed downstream to the DDC. The DDC will read the timestamp of the incoming CHDR packet, extract the IQ samples, and apply any queued frequency shift at the specified time in the stream. To get sample-precise data manipulation, the DDC can interpolate timestamps for each IQ sample by incrementing its sense of time by 1/samp_rate for each IQ sample.

With this method, a USRP can begin RX'ing data at absolute time t0, tune it's LO at an absolute time of t1, and then apply a frequency shift in the DDC at the point in data corresponding to t1, resulting in a physical and digital frequency change that begin on the exact same sample.

The reverse process holds true for the transmit chain. One often overlooked difference is that the host must pass along at least 1 sample with a timestamp included. Without this timestamped packet, the DUC can not derive a sense of time and therefore will never execute timed commands that are in its queue. This will either result in a command queue overflow or a "No Response Packet" runtime error from ctrl_iface.cpp.

Miscellaneous Timed Command Notes

Types of timed commands

  • Many timed commands amount to peeks and pokes to FPGA registers. Examples include gain changes, tune requests, etc.
  • Timed commands for transmitting data are a bit more involved, requiring queueing of samples, and arming of the device for a timed transmission.

Timed commands on AD93xx-based devices

  • Support for timed commands on AD93xx devices has been added.
    • It does not allow for phase resync of the AD93xx internal LO.
    • All non-Radio Core blocks all operate as you'd expect on USRPs with a dedicated daughterboard.
  • Timed commands for TX and RX on AD936x devices are supported, with caveats:
    • There will be a delay between an absolute time passing and the AD936x actually beginning an operation (TX/RX/Gain/Tune). This delay is deterministic if characterized for a given command.
    • SPI communication between the FPGA and RFIC means that the delay between an absolute time passing varies depending on the timed command issued.
    • This is to say that a timed command to begin TX and RX at the same time, on an AD936x-based device (which is in loopback) should result in a deterministic delay between TX and RX down to the sample.

Summary

Modern USRPs pass packets using the CHDR protocol. These packets can be used to issue commands to the USRP and may have an associated timestamp. A command with an included timestamp is called a timed command and it's important to understand how the USRP handles these timed commands. All blocks in a USRP's FPGA have a command queue and maintain a sense of time, however the Radio Core has the unique ability to store an absolute sense of time known as as the vita_time. When timed commands are issued to the USRP, they are added to a command FIFO of finite depth and are executed when the timestamp in the header of the command packet is >= the RFNoC block's sense of time. Utilizing these timed commands correctly allows for various USRP functionality to be executed with nanosecond precision, enabling time and phase coherent operation across multiple devices.

UHD API for Timed Commands

Overview of Relevant Methods

Commands that can be timed

Time Resolution of Timed Commands

Example: Using Timed Commands to Phase Align Channels

Example: Using Timed Commands to Control GPIO