Getting Started with RFNoC Development

Application Note Number

AN-823

Revision History

Date	Author	Details
2016-07-12	Martin Braun Nicolas Cuervo	Initial creation

Abstract

This application note gives a brief introduction into the steps required to start developing RFNoC blocks on your computer.

Overview

First, we are going to make sure that we have all the tools required to do RFNoC development. Then, we will set up a development sandbox in which you can do your RFNoC development. Finally, we will run through the basic steps required to actually write custom RFNoC blocks.

Prerequisites

For any of this to work, you need to make sure you have all the tools to do both software and FPGA development. Confirm you have the following tools installed:

• Xilinx Vivado: The specific version depends on the branch and state of the FPGA code and is very important. Make sure to install the correct version, you can look up the required version on the local version of the manual.
• Software build tools: If you can compile UHD from source, you have all the necessary components (If you're using PyBOMBS, this is definitely the case).

Please make sure these tools are available before proceeding to the next step.

Creating a development environment

In the following, we will assume that GNU Radio is used as a helper tool for RFNoC development. Note that GNU Radio is by no means required to use or develop RFNoC, but it makes it a great deal easier to use a framework on top of RFNoC for aspects such as visualization, among other features that ease the usage of this framework. GNU Radio is freely available and well suited for the task.

In order to get your development environment set up, you will need the following software packages:

• UHD
• GNU Radio
• gr-ettus

The cleanest way to set this up is to install everything into a dedicated directory. PyBOMBS is the simplest way to do this. If not already installed, PyBOMBS can be setup with the following steps:

First, make sure that the Python package installation tool pip is installed (the 'python-pip' package in Ubuntu and Fedora distributions). Then run the commands to download and configure PyBOMBS for RFNoC.

   $ sudo pip install git+https://github.com/gnuradio/pybombs.git
   $ pybombs recipes add gr-recipes git+https://github.com/gnuradio/gr-recipes.git
   $ pybombs recipes add ettus git+https://github.com/EttusResearch/ettus-pybombs.git

From here, the easiest way to set up a PyBOMBS development environment is to run the following command:

   $ pybombs prefix init ~/rfnoc -R rfnoc [-a alias]

This will do the following:

• Create a directory in ~/rfnoc (you can choose a different directory, of course) and give it an alias, which would be the name you give to this path. This name will be used in further steps that use PyBOMBS. When creating the first prefix and omitting the alias, the prefix will be setup as the default one in your system.
• Clone UHD, FPGA, GNU Radio, and gr-ettus sources into that directory
• Compile and install all the software

Then, it is necessary to setup the PyBOMBS environment, so that your system will have the environmental variables pointing to this newly created prefix:

   $ cd ~/rfnoc
   $ source ./setup_env.sh

PyBOMBS is by no means a requirement for doing any of these steps. If you wish to install the software manually, follow the individual install notes for GNU Radio, UHD and gr-ettus and make sure they are reachable by linkers and compilers.

To manually download the software, use these git clone commands, which will select the correct branches:

   $ git clone --recursive -b rfnoc-devel https://github.com/EttusResearch/uhd.git
   $ git clone --recursive -b maint https://github.com/gnuradio/gnuradio.git # master branch is also fine
   $ git clone -b radio-redo https://github.com/EttusResearch/gr-ettus.git
   $ git clone -b rfnoc-devel https://github.com/EttusResearch/fpga.git

If you already have UHD, GNU Radio and/or gr-ettus installed, it would be sufficient to checkout the branches mentioned and update them them (git pull). Thereafter, rebuild each of the repositories (first UHD, then GNU Radio, then gr-ettus).

NOTE: In some distributions (i.e. Ubuntu) dash is set as default shell, which may cause some issues. Given this case, it is recommended to set the shell to bash. You can do this by running:

   $ sudo dpkg-reconfigure dash

in your terminal, and choosing 'No' at the prompted dialogue.

Getting the default FPGA image

Of course, the whole point of RFNoC is to create your own FPGA images. However, we recommend you start with our default image, which is pre-built with a set of blocks (FFT, an FIR, etc.). To get this, simply run

   $ uhd_images_downloader

as you would with your standard source build. We will be updating the default image occasionally, and you can run uhd_images_downloader again after running git pull and re-installing. The usrp_x310_fpga_HG.bit file is modified with respect to master branch's version.

Next, you need flash the image to your device. You can either run uhd_image_loader with your usual settings (it will select the default RFNoC image if you downloaded them as discussed before), which will permanently install the image on the device (i.e., it will always be available after powering on and off).

Alternatively, you can simply JTAG the image to the FPGA using the viv_jtag_program utility, which is available after setting up the Vivado environment, which can be done by sourcing the setupenv.sh file located at {fpga-repository}/usrp3/top/e300/setupenv.sh. See the --help output for more information.

Testing the installation

OK, you now have everything in place to start hacking! If you have followed all the steps above, you now have:

• UHD/RFNoC downloaded and installed
• FPGA code available
• A valid RFNoC image on your X300/X310

   $ uhd_usrp_probe --init-only

If everything worked, you will see a lot of output (The RFNoC code is currently very verbose). The final lines of the output should look something like this:

Before continuing, run this command to see if everything is in order:

-- ========== Full list of RFNoC blocks: ============
-- * 0/Radio_0
-- * 0/Radio_1
-- * 0/DDC_0
-- * 0/DDC_1
-- * 0/DUC_0
-- * 0/DUC_1

Note: The actual names and number of blocks can differ. However, it should start with the radios, and then a couple more lines of block IDs should follow.

Getting started with UHD development

A good place to start looking how to use RFNoC from UHD is the examples directory. There are a few new examples:

• rx_samples_to_file can be configured to put an RFNoC block in between radio and host.
• rfnoc_nullsource_ce_rx Is an example that chains a null source to another block and streams the data to the host.

These examples demonstrate a lot of the core features of RFNoC. For further insights on UHD development please refer to the knowledge base or directly to the user manual

Getting started with GNU Radio + RFNoC

A good way of getting started with RFNoC in a more tangible fashion is to use GNU Radio. You will need the gr-ettus out-of-tree module (OOT) to be able to use RFNoC blocks in your local GNU Radio / GRC installation. This OOT contains a few blocks that allow you to configure your FPGA through GRC (Note: As these blocks mature, they will be upstreamed to gr-uhd).

Please note: [https://github.com/EttusResearch/gr-ettus gr-ettus] is a container used by Ettus Research to disseminate experimenta or under-development features for gr-uhd. It is not a replacement for gr-uhd (in fact, the latter is a requirement for gr-ettus).

After installation, you can run the examples from gr-ettus/examples/rfnoc.

A couple of rules for building GNU Radio flow graphs:

• You always need a device3 object in your flow graph (it does not get connected).
• You should have at least two RFNoC blocks connected together, going GNU Radio Block-> RFNoC Block -> GNU Radio Block is not recommended (it will work, but with suboptimal performance).

For more information on GNURadio development please refer to the GNURadio user's manual and API

RFNoC Framework

Before going any further, let's review briefly the RFNoC framework structure, whose understanding will help you considerably on your custom development. The colorful image down here shows the basic structure of the RFNoC Stack.

When diving into RFNoC development, the best way to do so, for convention, is by generating a so-called Out-Of-Tree (OOT) module, just as gr-ettus . From the stack there are clearly 5 different elements which a RFNoC developer has to take into account when creating an out-of-tree block. How these elements interact with each other is explained in more detail at [2][3][4]. However, from here we could easily infer that there are, at least, 3 components our out-of-tree block needs to address:

1. Does it have a complete FPGA Integration?

A Verilog file that describes the hardware implementation is what allows us to take advantage of the powerful FPGA that is being shipped with the current USRPs (and that will be shipped with the upcoming next-gen USRPs). For the current stage of development, the verilog_file.v file has to not only implement the custom desired functionality but also a efficient interaction with the AXI bus [8] and other FPGA blocks. In addition, the file has to be located in an specific location so that the building process can find it and use it as a resource for the IP generation.

2. Does it have a block declaration/control?

Now that the design has the hardware implementation in Verilog, we would like to take the samples (or other data items) out of the USRP and receive them at our host. The block has to have a block declaration file which defines, at least, its inputs and outputs and the NoC-ID, which is the method of identification between the HW part of our design and the SW part. Now, we might add some control over the samples that are being streamed into the block, and for that we have two options that depend on the complexity of the control that we want to apply: if as developers we can add control using NoC-Script, that could easily be added into the XML file. But if we need more exhaustive control and multiple data stream treatment, we might want to add more coding into it. Block controllers can be added as C++ in the UHD host, carefully generating the headers and adding them in the correct include folder and in the same manner, develop our control and add it in the library folder.

3. Does it have GNU Radio integration?

(Note: This is only required if using GNU Radio. Other frameworks might have similar requirements). GNU Radio is where the application runs, and where more control over the host is applied, by adding block-specific settings, such as sample rate or bandwidths. In addition, is where we have the our block ready to connect it to other several blocks (which can be also RFNoC blocks). That means that we are almost done with our design. In order to do so, we have to add a new XML file that allows us to bind our design with GNU Radio, and where we can edit the type on our I/Os, add variables for posterior signal processing, and a bunch of other design-specific parameters. In the same XML file we could add, just as we could have done at the block controllers, more block control by the means of NoC-Script. If we choose to do that, we could point our design to a "generic" implementation to control our GRC block. If we, otherwise, need even more control, we could choose to generate block code, that afterwards has to be located at gr-ettus: the headers as an include, the implementations as libraries.

So, you may think that this is just normal protocol of the RFNoC OOT block generation method. But, in order to answer this three questions with your OOT design, you have to take care of correct FPGA interfacing, correct API instantiation and right includes and correct location where to put your files... while you also put effort to the development of your custom design! Here at Ettus, we think that you should put all your effort in your custom design and let the RFNoC modtool do the dirty work.

Building your own custom block - RFNoC Modtool

In order to minimize the effort needed to implement a new RFNoC module, this gr_modtool-based tool provides the basic structure of the necessary files needed for the user to start with the development of his/her custom design right away. As it is gr_modtool-based, the manual found at [1] is also a great reference for this tool. Here, we discuss the features specific for the RFNoC modtool.

With the use of RFNoC ModTool, you can skip the struggle of locating the files of your design at different locations that depend (until now) on the location of your UHD and gr-ettus installation paths. Instead, you focus on your implementation and the tool takes care of boilerplate code and linking.

With the use of this tool, the questions stated in the last section would result in something like:

1. Does it have a complete FPGA Integration?

If not, start your design in the pre-generated verilog file, which also provides most of the standard AXI interface already done.

2. Does it have a block declaration/control?

Well, it has the files now, so write your functions if you need them. Don't even worry about modifying the files to add your NoC-ID: the tool will do that for you with the ID of your choice.

3 Does it have GNU Radio integration?

It has it now. You just need to put additional control, in case you need it.

Don't worry about relocating your files, they are where they need to be: in your custom location. The tool generates the files so that the whole building process takes into account your design regardless of the location of your OOT module in your machine.

Tool Installation

The tool is currently shipped within gr-ettus (radio-redo branch), so by installing gr-ettus, rfnocmodtool automatically becomes available.

Tool Utilization

After the installation, you are ready to use the tool to generate standard code for your Out-of-tree module. The fore-coming console outputs may vary depending on the version of UHD that you are running. However, functionality should be kept the same or closely similar. To check the usage of the tool, just type:

   $ rfnocmodtool help
   
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   Usage:
   rfnocmodtool <command> [options] -- Run <command> with the given options.
   rfnocmodtool help -- Show a list of commands.
   rfnocmodtool help <command> -- Shows the help for a given command. 
   
   List of possible commands:
   
   Name      Aliases          Description
   =====================================================================
   disable   dis              Disable block (comments out CMake entries for files) 
   info      getinfo,inf      Return information about a given module 
   remove    rm,del           Remove block (delete files and remove Makefile entries) 
   makexml   mx               Make XML file for GRC block bindings 
   add       insert           Add block to the out-of-tree module. 
   newmod    nm,create        Create a new out-of-tree module 
   rename    mv               Rename a block in the out-of-tree module. 

If you have used the gr_modtool provided by GNURadio, you'll be familiar with this usage.

Creating an RFNoC OOT Module

At this point you are all set up to start generating your own RFNoC OOT and if you were thinking that it is just as easy as it is with gr_modtool, you are completely right! Simply go to your prefixes source location (~/prefix/rfnoc/src) and type:

   $ rfnocmodtool newmod [NAME OF THE MODULE]

In the following, assume we want to create a module with the name 'example'. You can write the name of your module right after the 'newmod' command, but if you happen to forget it, the tool will ask for it interactively. This will create a folder containing the basic folders that you may need for a functional module.

   $ rfnocmodtool newmod example
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   
   Creating out-of-tree module in ./rfnoc-example... Done.
   Use 'rfnocmodtool add' to add a new block to this currently empty module.
   
   $ ls rfnoc-example/
   $ apps  cmake  CMakeLists.txt  docs  examples  grc  include  lib  MANIFEST.md  python  README.md  rfnoc  swig

In contrast with gr_modtool, this includes a folder called 'rfnoc', which is where we put the UHD/FPGA files (more details below).

Adding custom blocks

Now you can start adding blocks to your module. The only thing to do is go inside of the module you just created and use the add command of rfnocmodtool:

   $ cd rfnoc-example
   $ rfnocmodtool add [NAME OF THE BLOCK]

For demonstrative purposes, lets create a block named 'copy'. As for the new module, the tool will ask you for the name of the block if you not put it in the first line. There is a bunch of directives that you could pass to the tool, but if you run it without any of these directives you should expect the following interactive parsing output:

   $ rfnocmodtool add copy
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   
   RFNoC module name identified: example
   Block/code identifier: copy
   Enter valid argument list, including default arguments: 
   Add Python QA code? [Y/n] 
   Add C++ QA code? [y/N] 
   Block NoC ID (Hexadecimal): FFFF
   Skip Block Controllers Generation? [y/N] 
   Skip Block interface Generation? [y/N] 

The tool is going to ask you about a valid argument list, in case you have that clear at this stage of your design, and for the generation of QA codes which are, such as in the case of gr_modtool, optional. Right after that, specific RFNoC questions appear:

• NoC ID: This ID is a Hexadecimal number which serves as identification between the hardware part and the software part of the design. It can be as long as 16 0-9 A-F digits. If you don't provide the NoC ID, it will be set to a random number.
• Block Controllers Generation: The block controllers are the C++ control that you can apply to the UHD-part of the design. In these files, you can add more control over this layer of your design. However, if you can manage to add the control you need over NoC-Script, you can omit the cpp/hpp block control files generation, reason why this is optional.
• Block Interface: Analogously to the block controllers, you may want to add more design specific functionality to your design at the GNURadio interface. To do that, you add the logic into the block-interface files. In the same manner, if your control can be managed by adding NoC-Script to your xml file, you can disregard the generation of files.

Also, it is important to notice that if you don't intend to use the block controllers or you are not sure if you would, the presence of them in your design would do no harm. So, if in doubt, just add them. You may or may not add more functions inside them in a future stage of development of your OOT module. After finishing the parsing, the following files are going to be generated/edited:

   Adding file 'lib/copy_impl.h'...
   Adding file 'lib/copy_impl.cc'...
   Adding file 'include/example/copy.h'...
   Adding file 'include/example/copy_block_ctrl.hpp'...
   Adding file 'lib/copy_block_ctrl_impl.cpp'...
   Editing swig/example_swig.i...
   Adding file 'python/qa_copy.py'...
   Editing python/CMakeLists.txt...
   Adding file 'grc/example_copy.xml'...
   Adding file 'rfnoc/blocks/copy.xml'...
   Adding file 'rfnoc/fpga-src/noc_block_copy.v'...
   rfnoc/testbenches/noc_block_copy_tb folder created
   Adding file 'rfnoc/testbenches/noc_block_copy_tb/noc_block_copy_tb.sv'...
   Adding file 'rfnoc/testbenches/noc_block_copy_tb/Makefile'...
   Adding file 'rfnoc/testbenches/noc_block_copy_tb/CMakeLists.txt'...

And, at this stage, you are all set up to start adding your custom functionalities.

Writing Computation Engines

NoC Shell

RFNoC blocks or Computation Engines (CEs) in the FPGA use a NoC Shell instance to interface with the rest of RFNoC. NoC Shell implements RFNoC's core functionality: packet muxing and demuxing, flow control, and the settings register bus (i.e. write/read control/status registers). The NoC Shell has an interface to the RFNoC AXI stream crossbar and a user interface.

The user interface consists of four AXI stream interfaces and a settings register bus:

• str_sink: Input sample data packets
  • Data coming from host or another CE
• str_src: Output sample data packets
  • Data going to another CE or host
• cmdout: Command packets
  • Typically used to write settings registers in another CE
• ackin: Acknowledge packets
  • Responses from command packet, contains value from selected readback register
  • For register readback, send command packet to set readback address and acknowledge packet will have readback register value
• set_data/addr: Settings register bus that provides write/read access to control/status registers

Note: NoC Shell AXI stream interfaces expect CHDR packets with a proper header! See the manual for information on CHDR and SID.

AXI Stream is an ARM AMBA standard interface. Xilinx has an AXI Reference Guide with more details on this standard.

AXI Wrapper

Many designs will want to use an AXI Stream interface with only sample data. However, the NoC Shell expects CHDR packets. To ease interfacing user code, the AXI Wrapper block provides the necessary logic to strip and insert the CHDR header, effectively converting packetized sample data into streaming sample data and vice versa. The example RFNoC blocks noc_block_fft.v and noc_block_fir.v show how AXI Wrapper is used to implement existing Xilinx AXI Stream based IP within a computation engine.

Note: AXI Wrapper also supports AXI Stream buses for configuration. These buses are driven via the setting register bus and do not have back pressure. They also consume two user register addresses per bus.

Wiring up computation engines and building the FPGA image

Using the command line

After you are done with your block development, you may want to generate the fpga image to actually run your implementation. A small script to help you do this is provided within the fpga repository that you installed in the Creating a development environment section.

The script is called 'make.py', and it is located at {INSTALL_PREFIX}/src/uhd-fpga/usrp3_rfnoc/tools/scripts. You can run the help by typing:

   $ ./make.py --help
   
   usage: make.py [-h] [-I INCLUDE_DIR [INCLUDE_DIR ...]] [-m MAX_NUM_BLOCKS]
              [--fill-with-fifos] [-o OUTFILE] [-d DEVICE] [-t TARGET]
              [blocks [blocks ...]]
   
   Generate the NoC block instantiation file
   
   positional arguments:
     blocks                List block names to instantiate.
   
   optional arguments:
     -h, --help            show this help message and exit
     -I INCLUDE_DIR [INCLUDE_DIR ...], --include-dir INCLUDE_DIR [INCLUDE_DIR ...]
                           Path directory of the RFNoC Out-of-Tree module
     -m MAX_NUM_BLOCKS, --max-num-blocks MAX_NUM_BLOCKS
                           Maximum number of blocks (Max. Allowed for x310|x300:
                           10, for e300: 6)
     --fill-with-fifos     If the number of blocks provided was smaller than the
                           max number, fill the rest with FIFOs
     -o OUTFILE, --outfile OUTFILE
                           Output /path/filename - By running this directive, you
                           won't build your IP
     -d DEVICE, --device DEVICE
                           Device to be programmed [x300, x310, e310]
     -t TARGET, --target TARGET
                           Build target - image type [X3X0_RFNOC_HG,
                           X3X0_RFNOC_XG, E310_RFNOC_sg3...]

There are a fair amount of arguments that you can pass to this script. For example, let's create an FPGA image using the 'copy' block we created earlier:

   make.py fft copy fft window -I {OOT_moddir/rfnoc-example}/rfnoc/fpga-src/ -d x310 -t X310_RFNOC_HG -m 5 --fill-with-fifos [-o rfnoc_inst.v]

We're also adding an FFT and Window block, which are already in the script's path (they ship with the the Ettus FPGA code).

Here's how the script is used:

• Blocks: The first arguments are the names of blocks that you want to have in your image separated by a space. They can be from the blocks that you have from your OOT module or from the ones that are provided from Ettus, or a combination of those as well. The script is going to be looking for the FPGA name of the RFNoC blocks which, in the case of the blocks provided by Ettus, are listed (among another sources necessary for the FPGA build) at lib/rfnoc/Makefile.srcs file in the usrp3 folder of your FPGA repository. This blocks are called "noc_block_{FOO}.v". However, as all the RFNoC blocks have the same "noc_block_" prefix, for simplicity this prefix is taken for granted while listing the blocks you want to add while using the make.py script. I.e. if you want to add a null sink/source block and a signal generator, instead of writing

   $ ./make.py noc_block_null_source_sink noc_block_siggen # Wrong

you should write

   $ ./make.py null_source_sink siggen                     # Right

Which doesn't seem like a big change, but saves a significant amount of time/typing in the long run. It is important to notice that the blocks generated by the RFNoCModtool follow the same naming convention.

We have an increasing list of provided blocks (without taking into account your own blocks), but some of the block names that you can pass directly to make.py are:

• axi_fifo_loopback
• axi_dma_fifo
• fir_filter
• fft
• null_source_sink
• schmidl_cox
• packet_resizer
• split_stream
• vector_iir
• addsub
• window
• keep_one_in_n
• pfb
• export_io
• conv_encoder_qpsk
• siggen
• logpwr
• fosphor
• moving_avg
• ddc
• duc

Keep in mind that, so far, there is a restriction on the amount of blocks that you can add into the FPGA image: 10 for the X3X0 devices, and 14 for the E3X0 (as long as the space in the FPGA permits).

• Include_dir: To add your custom block to the FPGA Image, you just have to tell the Vivado builder where to find it. Here, with the -I directive you give the /Path/to/your/fpga-src, which is inside the rfnoc/ folder of your RFNoC OOT module. It is important to notice that inside this folder there is also a file called 'Makefile.srcs', in which you can find the path of your OOT and a list of all the OOT blocks that you have developed in your custom module. This is an autogenerated file, which is populated everytime that you add a block. Manually modifying this block may result in inconveniences.

Providing this path is non compulsory, so you can use this script to kick off a FPGA build with ease even if you intend to add only Ettus-provided blocks. Also, if you want to add more than one OOT module, you can add the different paths separating them by a space (eg. -I /first/OOT/path/ /second/OOT/path/)

• Device: Generation-3 USRP devices support RFNoC, but the nextGen USRPs will support RFNoC as well. Having this in mind, with this -d directive you tell the script in which device you intend to use the image. By omitting this directive, the script will choose the X310 device as default. In the example, however, we chose explicitly the -d x310

• Target: there are several build options, and you can choose the one that suits better with your design with the -t directive. Detailed information about the build targets can be found at the build instructions for USRP3, found under [6]. If -t is not given, a default target will be chosen for the given device. In the example, the -t X310_RFNOC_HG builds our design intended for the USRP X310 device, with 1GigE on SFP+ Port0, 10Gig on SFP+ Port1, and a DRAM TX FIFO.

• amount of blocks: if your RFNoC image doesn't really have the need of several RFNoC blocks, you can set the number of blocks to be inserted in the FPGA by running the -m directive. In the example command, 5 blocks are being added to the .bit file.

• Fill with FIFOs: You can list a number of blocks smaller than the maximum allowed (or than the number fixed with -m). In addition, you can fill the rest of free slots with FIFO blocks. In the example, one FIFO block is going to be added to the FPGA image.

• Output file: the purpose of the whole 'make.py' script is to autogenerate an instantiation file and populate the source files needed for Vivado to build the FPGA satisfactorily. But sometimes you may only want to see the effect of adding your OOT module on the fpga/ directory, or how the instantiation file would look like. With the -o directive, you generate the instantiation file and save it at the desired path with the given name. In the example, it is saved at the tmp/ directory with the name verilog_CE.v. Keep in mind that if you don't give the -o directory, the rfnoc_ce_auto_inst_x3x0.v file will be overwritten and the FPGA image build process will start automatically.

The FPGA image building process may take over an hour.

NOTE: [Environment setup] - The script will also set up your Vivado environment by automatically running the 'setupenv.sh' located at /top/{device}. So you don't have to run this script before running the make.py. In addition, keep in mind that the setupenv.sh assumes that your Vivado is installed in the default location /opt/Xilinx/Vivado. If you chose a different installation directory, you may want to modify the setupenv_base.sh script, located inside the fpga directory at ./tools/scripts/setupenv_base.sh

Using a graphical interface

This feature will be part of one of the upcoming releases

Now, if you are not very fond to th--helpe use of command line scripts, a graphical user interface for FPGA generation and building is shipped along with the make.py. This GUI will help you set up the FPGA build that you need in an intuitive way. To run it, go to {fpga-repo}/usrp3_rfnoc/tools/scripts/ (which is where the make.py is also located), and simply run:

   $ ./GUI

Where this panel will be launched:

Although the functionality of the elements that compose the interface is rather easy to infer, a brief explanation of each is as follows:

1. Select build target: in this panel the available build targets are listed. This list may vary depending on which branch of the FPGA repository you are using for your development. In addition, although Vivado can build non-RFNoC images, those are not intended to be customized, reason why they are not listed here. Important to notice is a clear pattern on the elements of this list, which follows {DEVICE}_RFNOC_{BUILD_TYPE}, being {DEVICE} the currently USRP’s that support RFNoC, and the build type being as follows:

    HG: 1GigE on SFP+ Port0, 10Gig on SFP+ Port1
    XG: 10GigE on both SFP+ ports
    HLS: Vivado High Level Synthesis enabled
    sgX: Speed grade for E300 devices (1 or 3)

2. List of blocks available: in this panel the blocks available to be included into a custom design are listed. This list separates the RFNoC blocks provided by Ettus and the eventual Out-of-tree modules that the user adds to the libraries. This is helpful to have at hand the complete list of sources that can be added into the FPGA. Keep in mind that, given the differences under-the-hood between the X300 and E300 devices, this list will dynamically change when a different device is selected from the panel on the left. This also means that it is *necessary* to add the OOT modules for each device independently. How to do this with the GUI is explained at the numeral 7.

3. Blocks in current design: This panel will be populated by adding elements from the available blocks. All the blocks listed in here will be added into the FPGA custom image. However there is a maximum of blocks that can be added for each device. The maximum amount of computation engines that each device can use is 16, but the amount of custom blocks that can be added depends on the device: If you plan to use a device from the X300 family, from this 16 CE, there are 6 that will be always added and are not subject to direct customization: 1 CE for the AXI bus, 1 CE for the Ethernet Interface, 2 Radios and 2 Dma FIFOS. Because of this, the GUI will only allow a number of 10 custom blocks. If you are using a device from the E300 family, 2 CE engines are always added and are not subject to direct customization: 1 CE for the AXI bus and 1 Radio. This would virtually allow 14 slots for custom blocks. However, given the size of the FPGA on this family of devices, the GUI only allows a number of 6 custom blocks. NOTE: This number of blocks is just a rough estimate! Although this # of blocks suffice most of the user cases, custom blocks with higher utilization may fill up the FPGA and force the user to include less custom blocks.

4. Add button (>>): With this button you manually add the blocks from the central panel into your design.

5. Remove button (<<): Similarly, with this button you remove blocks from the current design (far-left panel)

6. Fill with FIFOs: when just a fairly less amount of blocks are needed for certain implementation, many users like to fill their design with FIFO blocks. By clicking this box, your design will do so under the rules of amount of blocks explained at #3.

7. Add OOT blocks: With this button you can manually add your RFNoCModtool-generated OOT modules by pointing the GUI to the Makefile.srcs, which is located at {your-OOT-moddir}/rfnoc/fpga-srcs/Makefile.srcs. After adding this file, your blocks should appear under “OOT blocks for XXXX devices”

8. Import from GRC: if you rather first set up a flowgraph in GNU Radio companion, or you are migrating from a purely GRC implementation by adding some RFNoC blocks, you can pull your flowgraph into the GUI and it will list the RFNoC blocks needed for this design into the far-right panel. Keep in mind that if some of the elements pulled from a .grc file has not yet been added into the available blocks (central panel) you’d have to do so before kicking off the build.

9. Show Instantiation File: This GUI auto-generates the instantiation file that is going to be used by Vivado to build the FPGA image. If you fancy have a glance into this file you might do so by clicking on this button. You can also edit this file to your convenience before starting the build.

10. Generate .bit file: when you are ready with your setup, you can start the build by clicking on this button. As well as the make.py script, this button will set up your Vivado environment and proceed to build the .bit image based on your choices.

Creating and running testbenches

In order to make the iteration process much quicker, we encourage you to design test benches for each of your RFNoC blocks before implementing them into the FPGA image. By doing so, you can detect bugs or flaws in a relatively early stage of your design. RFNoC modtool provides the structure for the testbenches for each of the out-of-tree blocks that you add with the "rfnocmodtool add" option.

You can see that in the Adding custom blocks section, the last files generated are:

  rfnoc/testbenches/noc_block_copy_tb folder created
  Adding file 'rfnoc/testbenches/noc_block_copy_tb/noc_block_copy_tb.sv'...
  Adding file 'rfnoc/testbenches/noc_block_copy_tb/Makefile'...
  Adding file 'rfnoc/testbenches/noc_block_copy_tb/CMakeLists.txt'...

The noc_block_copy_tb is a folder generated to contain all the files related to the test bench of the copy block. Each time that you create a new OOT block, a new folder will be generated as well. Inside of this folder, three files are located:

• CMakeLists.txt: this is an empty file used, so far, only to increase the scope of the compilers.
• noc_block_copy_tb.sv: this is a System Verilog file, in which your custom tests are to be located. This is the only file that you have to modify according with your design.
• Makefile: This file determines the directives that run the simulation.

This files require the Vivado tools to work, so you have to have this tool installed (preferably at the default location, /opt/Xilinx). Also, you would have to set up the Vivado environmental variables. Until now, you had to do this by hand, which is somewhat cumbersome. Now, with the use of the structure given by RFNoC modtool, the only thing that you have to do is build your own OOT module, which you have to do anyways! The builders will look up for the default locations of Vivado and the fpga repository and will set up your test benches so that they are ready to use, and you can fully focus on your design. To do so, simply create a build directory:

   mkdir build && cd build/

and then run cmake. If you are using PyBOMBS, cmake will detect automatically the location of the fpga repository. If you are installing by source, you'd have to provide the location where you installed it.

   cmake [-DUHD_FPGA_DIR=/PATH/TO/FPGA/REPOSITORY]

Then, and only for the first time, you have to run the following command:

   make test_tb

This command will modify your files by setting the correct path to the simulation tools. From now on, every time that you add a new block, this command will be run automatically.

Now you are ready to run the test bench! you don't have to go to the testbench directory, you can run it right there at the OOT main 'build/' directory! just run make noc_block_[name_of_your_block]_tb, which in this example case is:

   make noc_block_copy_tb

And the simulation will start. With every block that you create, also a make directive will be available for you to run the simulation right there at the main build directory.

NOTE: In some of the latests versions of UHD there was a distinction between the FPGA files destinated to RFNoC and those who weren't. This was made by creating a new folder within the FPGA repository called usrp3_rfnoc:

   ~/src/rfnoc/src/uhd-fpga$ ls
   
   docs  README.md  usrp1  usrp2  usrp3  usrp3_rfnoc

If you are working with some of the latests versions of our software, you may have noticed this new folder. This folder is meant to be temporal, and is going away rather soon. However, the files generated by rfnocmodtool use the former path to look for the requiered files necessary for the test bench. Being this the case, you may run into something like this while trying to run it from the build folder:

   make noc_block_copy_tb

   ...
   # if [expr ([string equal $simulator "XSim"] == 0) && ([string length $sim_complibdir] == 0)] {
   #     puts "BUILDER: \[ERROR\]: Could not resolve the location for the compiled simulation libraries."
   #     puts "                  Please build libraries for chosen simulator and set the env or"
   #     puts "                  makefile variable SIM_COMPLIBDIR to point to the location."
   #     exit 1
   # }
   # puts "BUILDER: Creating Vivado simulation project part $part_name"
   BUILDER: Creating Vivado simulation project part 
   # create_project -part $part_name -force $project_name/$project_name
   WARNING: [#UNDEF] No parts matched 
   'ERROR: [Coretcl 2-106] Specified part could not be found.'
   INFO: [Common 17-206] Exiting Vivado at Tue Nov  8 18:54:20 2016...
   make[4]: *** [xsim] Error 1
   make[3]: *** [CMakeFiles/noc_block_copy_tb] Error 2
   make[2]: *** [CMakeFiles/noc_block_copy_tb.dir/all] Error 2
   make[1]: *** [CMakeFiles/noc_block_copy_tb.dir/rule] Error 2
   make: *** [noc_block_copy_tb] Error 2

And this is caused only because rfnocmodtool still thinks that all you need is in the folder usrp3. The workaround is rather simple. Simply go to the Makefile of your testbench and modify the basepath that is there so that it points to the right folder. So, in the Makefile, turn:

   BASE_DIR = {your_prefix}/src/uhd-fpga/usrp3/top 

into

   BASE_DIR = {your_prefix}/src/uhd-fpga/usrp3_rfnoc/top 

And just by doing so, your testbench should run without inconveniences in the build directory.

Other Resources

[1] GNURadio OutOfTree Modules tutorial

[2] Introduction to RFNoC

[3] RFNoC Deep Dive: FPGA

[4] RFNoC Deep Dive: Host side

[5] Video: RFNoC presented at Wireless @ Virginia Tech, 2015 Explaining the slides of [2][3][4] (In that order)

[6] USRP3 build instructions

[7] Knowledge base guide: RFNoC getting started

[8] Xilinx - AXI reference guide

[9] RFNoC Main KB Article