Application Note Number

AN-823

Revision History

Date	Author	Details
2016-07-12	Martin Braun Nicolas Cuervo	Initial creation

Abstract

This application note gives a brief introduction into the steps required to start developing RFNoC blocks on your computer.

Overview

First, you need to make sure that you have all the tools required to do RFNoC development. Then, we will set up a development sandbox in which you can do your RFNoC development. Finally, we will run through the basic steps required to actually write custom RFNoC blocks.

Prerequisites

For any of this to work, you need to make sure you have all the tools to do both software and FPGA development. Confirm you have the following tools installed:

Xilinx Vivado. The specific version depends on the branch and state of the FPGA code and is very important. Make sure to install the correct version, you can look up the required version on the local version of the manual.
Software build tools. If you can compile UHD from source, you have all the necessary components (If you're using PyBOMBS, this is definitely the case).

Please make sure these tools are available before proceeding to the next step.

Creating a development environment

In the following, we will assume that GNU Radio is used as a helper tool for RFNoC development. Note that GNU Radio is by no means required to use or develop RFNoC, but it makes it a great deal easier to use a framework on top of RFNoC for visualization etc., and GNU Radio is freely available and well suited for the task. In order to get your development environment set up, you will need the following software packages:

UHD
GNU Radio
gr-ettus

The cleanest way to set this up is to install everything into a dedicated directory. PyBOMBS is the simplest way to do this. If you haven't already downloaded and installed PyBOMBS, you can do so by running the following commands:

   $ sudo pip install git+https://github.com/gnuradio/pybombs.git
   $ pybombs recipes add gr-recipes git+https://github.com/gnuradio/gr-recipes.git
   $ pybombs recipes add ettus git+https://github.com/EttusResearch/ettus-pybombs.git

From here, the easiest way to set up a PyBOMBS development environment is to run the following command:

   $ pybombs prefix init ~/prefix/rfnoc -R rfnoc [-a alias]

This will do the following:

Create a directory in ~/prefix/rfnoc (you can choose a different directory, of course)
Clone UHD, FPGA, GNU Radio, and gr-ettus sources into that directory
Compile and install all the software

Then, you need to enable the prefix:

   $ cd ~/prefix/rfnoc
   $ source ./setup_env.sh

PyBOMBS is by no means a requirement for doing any of these steps. If you wish to install the software manually, follow the individual install notes for GNU Radio, UHD and gr-ettus and make sure they are reachable by linkers and compilers.

To manually download the software, use these git clone commands, which will select the correct branches:

   $ git clone --recursive -b rfnoc-radio-redo https://github.com/EttusResearch/uhd.git
   $ git clone --recursive -b maint https://github.com/gnuradio/gnuradio.git # master branch is also fine
   $ git clone -b radio-redo https://github.com/EttusResearch/gr-ettus.git

If you already have UHD, GNU Radio and/or gr-ettus installed, it would be sufficient to checkout the branches mentioned and update them them (git pull). Thereafter, rebuild each of the repositories (first UHD, then GNU Radio, then gr-ettus).

RFNoC Framework

First, let's review briefly the RFNoC framework structure. The colourful image down here shows the basic structure of the RFNoC Stack.

From the stack there are clearly 5 different elements which a RFNoC developer has to take into account when creating an out-of-tree block. How these elements interact with each other is explained in more detail at [2][3][4]. However, from here we could easily infer that there are, at least, 3 components our out-of-tree block needs to address:

1. Does it have a complete FPGA Integration?: A Verilog file that describes the hardware implementation is what allows us to take advantage of the powerful FPGA that is being shipped with the current and next gen USRPs. For the current stage of development, the .v file has to not only implement the custom desired functionality but also a efficient interaction with the AXI bus and other FPGA blocks. In addition, the file has to be located in an specific location so that the building process can find it and use it as a resource for the IP generation.

2. Does it have a block declaration/control?: Now that the design has the hardware implementation in Verilog, we would like to take the samples (or other data items) out of the USRP and receive them at our host. The block has to have a block declaration file which defines, at least, its inputs and outputs and the NoC-ID, which is the method of identification between the HW part of our design and the SW part. Now, we might add some control over the samples that are being streamed into the block, and for that we have two options that depend on the complexity of the control that we want to apply: if as developers we can add control using NoC-Script, that could easily be added into the XML file. But if we need more exhaustive control and multiple data stream treatment, we might want to add more coding into it. Block controllers can be added as C++ in the UHD host, carefully generating the headers and adding them in the correct include folder and in the same manner, develop our control and add it in the library folder.

3. Does it have GNU Radio integration?: (Note: This is only required if using GNU Radio. Other frameworks might have similar requirements). GNU Radio is where the application runs, and where more control over the host is applied, by adding block-specific settings, such as sample rate or bandwidths. In addition, is where we have the our block ready to connect it to other several blocks (which can be also RFNoC blocks). That means that we are almost done with our design. In order to do so, we have to add a new XML file that allows us to bind our design with GNU Radio, and where we can edit the type on our I/Os, add variables for posterior signal processing, and a bunch of other design-specific parameters. In the same XML file we could add, just as we could have done at the block controllers, more block control by the means of NoC-Script. If we choose to do that, we could point our design to a "generic" implementation to control our GRC block. If we, otherwise, need even more control, we could choose to generate block code, that afterwards has to be located at gr-ettus: the headers as an include, the implementations as libraries.

So, you may think that this is just normal protocol of the RFNoC OOT block generation method. But, in order to answer this three questions with your OOT design, you have to take care of correct FPGA interfacing, correct API instantiation and right includes and correct location where to put your files... while you also put effort to the development of your custom design! Here at Ettus, we think that you should put all your effort in your custom design and let the RFNoC modtool do the dirty work.

RFNoC Modtool

In order to minimize the effort needed to implement a new RFNoC module, this gr_modtool-based tool provides the basic structure of the necessary files needed for the user to start with the development of his/her custom design right away. As it is gr_modtool-based, the manual found at [1] is also a great reference for this tool. Here, we discuss the features specific for the RFNoC modtool.

With the use of RFNoC ModTool, you can skip the struggle of locating the files of your design at different locations that depend (until now) on the location of your UHD and gr-ettus installation paths. Instead, you focus on your implementation and the tool takes care of boilerplate code and linking.

With the use of this tool, the questions stated in the last section would result in something like:

1. Does it have a complete FPGA Integration?: If not, start your design in the pregenerated verilog file, which also provides most of the standard AXI interface already done.

2. Does it have a block declaration/control?: Well, it has the files now, so write your functions if you need them. Don't even worry about modifying the files to add your NoC-ID: the tool will do that for you with the ID of your choice.

3 Does it have GNU Radio integration?: It has it now. You just need to put additional control, in case you need it.

Don't worry about relocating your files, they are where they need to be: in your custom location. The tool generates the files so that the whole building process takes into account your design regardless of the location of your OOT module in your machine.

Tool Installation

The tool is currently shipped within gr-ettus (radio-redo branch), so by installing gr-ettus, rfnocmodtool automatically becomes available.

Tool Utilization

After the installation, you are ready to use the tool to generate standard code for your Out-of-tree module. To check the usage of the tool, just type:

   $ rfnocmodtool help
   
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   Usage:
   rfnocmodtool <command> [options] -- Run <command> with the given options.
   rfnocmodtool help -- Show a list of commands.
   rfnocmodtool help <command> -- Shows the help for a given command. 
   
   List of possible commands:
   
   Name      Aliases          Description
   =====================================================================
   disable   dis              Disable block (comments out CMake entries for files) 
   info      getinfo,inf      Return information about a given module 
   remove    rm,del           Remove block (delete files and remove Makefile entries) 
   makexml   mx               Make XML file for GRC block bindings 
   add       insert           Add block to the out-of-tree module. 
   newmod    nm,create        Create a new out-of-tree module 
   rename    mv               Rename a block in the out-of-tree module.

If you have used the gr_modtool provided by GNURadio, you'll be familiar with this usage.

Creating an RFNoC OOT Module

At this point you are all set up to start generating your own RFNoC OOT and if you were thinking that it is just as easy as it is with gr_modtool, you are completely right! Simply go to your prefixes source location (~/prefix/rfnoc/src) and type:

   $ rfnocmodtool newmod [NAME OF THE MODULE]

In the following, assume we want to create a module with the name 'example'. You can write the name of your module right after the 'newmod' command, but if you happen to forget it, the tool will ask for it interactively. This will create a folder containing the basic folders that you may need for a functional module.

   $ rfnocmodtool newmod example
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   
   Creating out-of-tree module in ./rfnoc-example... Done.
   Use 'rfnocmodtool add' to add a new block to this currently empty module.
   
   $ ls rfnoc-example/
   $ apps  cmake  CMakeLists.txt  docs  examples  grc  include  lib  MANIFEST.md  python  README.md  rfnoc  swig

In contrast with gr_modtool, this includes a folder called 'rfnoc', which is where we put the UHD/FPGA files (more details below).

Adding custom blocks

Now you can start adding blocks to your module. The only thing to do is go inside of the module you just created and type:

   $ rfnocmodule add [NAME OF THE BLOCK]

For demonstrative purposes, lets create a block named 'copy'. As for the new module, the tool will ask you for the name of the block if you not put it in the first line. There is a bunch of directives that you could pass to the tool, but if you run it without any of these directives you should expect the following interactive parsing output:

   $ rfnocmodtool add copy
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   
   RFNoC module name identified: example
   Block/code identifier: copy
   Enter valid argument list, including default arguments: 
   Add Python QA code? [Y/n] 
   Add C++ QA code? [y/N] 
   Block NoC ID (Hexadecimal): FFFF
   Skip Block Controllers Generation? [y/N] 
   Skip Block interface Generation? [y/N]

The tool is going to ask you about a valid argument list, in case you have that clear at this stage of your design, and for the generation of QA codes which are, such as in the case of gr_modtool, optional. Right after that, specific RFNoC questions appear:

NoC ID: This ID is a Hexadecimal number which serves as identification between the hardware part and the software part of the design. It can be as long as 16 0-9 A-F digits. If you don't provide the NoC ID, it will be set to a random number.
Block Controllers Generation: The block controllers are the C++ control that you can apply to the UHD-part of the design. In these files, you can add more control over this layer of your design. However, if you can manage to add the control you need over NoC-Script, you can omit the cpp/hpp block control files generation, reason why this is optional.
Block Interface: Analogously to the block controllers, you may want to add more design specific functionality to your design at the GNURadio interface. To do that, you add the logic into the block-interface files. In the same manner, if your control can be managed by adding NoC-Script to your xml file, you can disregard the generation of files.

Also, it is important to notice that if you don't intend to use the block controllers or you are not sure if you would, the presence of them in your design would do no harm. So, if in doubt, just add them. You may or may not add more functions inside them in a future stage of development of your OOT module. After finishing the parsing, the following files are going to be generated/edited:

   Adding file 'lib/copy_impl.h'...
   Adding file 'lib/copy_impl.cc'...
   Adding file 'include/example/copy.h'...
   Adding file 'include/example/copy_block_ctrl.hpp'...
   Adding file 'lib/copy_block_ctrl_impl.cpp'...
   Editing swig/example_swig.i...
   Adding file 'python/qa_copy.py'...
   Editing python/CMakeLists.txt...
   Adding file 'grc/example_copy.xml'...
   Adding file 'rfnoc/blocks/copy.xml'...
   Adding file 'rfnoc/fpga-src/noc_block_copy.v'...
   rfnoc/testbenches/noc_block_copy_tb folder created
   Adding file 'rfnoc/testbenches/noc_block_copy_tb/noc_block_copy_tb.sv'...
   Adding file 'rfnoc/testbenches/noc_block_copy_tb/Makefile'...
   Adding file 'rfnoc/testbenches/noc_block_copy_tb/CMakeLists.txt'...

And, at this stage, you are all set up to start adding your custom functionalities.

Building the FPGA image

After you are done with your block development, you may want to generate the fpga image to actually run your implementation. A small script to help you do this is provided within the fpga repository that you installed in the Creating a development environment section.

The script is called 'make.py', and it is located at {INSTALL_PREFIX}/fpga/usrp3/tools/scripts. You can run the help by typing:

   make.py --help
   
   Generate the NoC block instantiation file
   
   positional arguments:
     blocks                List block names to instantiate.
   
   optional arguments:
     -h, --help            show this help message and exit
     -I INCLUDE_DIR, --include-dir INCLUDE_DIR
                           Path directory of the RFNoC Out-of-Tree module fpga srcs
                           {OOT_moddir}/rfnoc/fpga-src
     -m MAX_NUM_BLOCKS, --max-num-blocks MAX_NUM_BLOCKS
                           Maximum number of blocks (Max. Allowed for x310|x300:
                           11, for e300:6)
     --fill-with-fifos     If the number of blocks provided was smaller than the
                           max number, fill the rest with FIFOs
     -o OUTFILE, --outfile OUTFILE
                           Output /path/filename - By running this directive, you
                           won't build your IP
     -d DEVICE, --device DEVICE
                           Device to be programmed [x300, x310, e310]
     -t TARGET, --target TARGET
                           Build target - image type [X3X0_RFNOC_HG,
                           X3X0_RFNOC_XG, E310_RFNOC_sg3...]

There are also a good amount of arguments that you can pass to this script. For example, let's create an FPGA image using the 'copy' block we created earlier:

   make.py fft copy fft window -I {OOT_moddir/rfnoc-example}/rfnoc/fpga-src/ -d x310 -t X310_RFNOC_HG -m 5 --fill-with-fifos [-o rfnoc_inst.v]

We're also adding an FFT and Window block, which are already in the script's path (they ship with the the Ettus FPGA code).

Here's how the script is used:

Blocks: The first arguments are the names of blocks that you want to have in your image separated by a space. They can be from the blocks that you have from your OOT module or from the ones that are provided from Ettus, or a combination of those as well. There are two things to keep in mind while listing blocks here:
- So far, there is a restriction on the amount of blocks that you can add into the FPGA image: 10 for the X3X0 devices, and 14 for the E3X0 (if space permits).

Although the amount of blocks is constantly increasing (thanks to the constant development and contribution of the community), a brief overview of the blocks from which you can choose can be found at [7].

Include_dir: To add your custom block to the FPGA Image, you just have to tell the Vivado builder where to find it. Here, with the -I directive you give the /Path/to/your/fpga-src, which is inside the rfnoc/ folder of your RFNoC OOT module. It is important to notice that inside this folder there is also a file called 'Makefile.srcs', in which you can find the path of your OOT and a list of all the OOT blocks that you have developed in your custom module. This is an autogenerated file, which is populated everytime that you add a block. Manually modifying this block may result in inconveniences.

Device: Generation-3 USRP devices support RFNoC, but the nextGen USRPs will support RFNoC as well. Having this in mind, with this -d directive you tell the script in which device you intend to use the image. By omitting this directive, the script will choose the X310 device as default. In the example, however, we chose explicitly the -d x310

Target: there are several build options, and you can choose the one that suits better with your design with the -t directive. Detailed information about the build targets can be found at the build instructions for USRP3, found under [6]. If -t is not given, a default target will be chosen for the given device. In the example, the -t X310_RFNOC_HG builds our design intended for the USRP X310 device, with 1GigE on SFP+ Port0, 10Gig on SFP+ Port1, and a DRAM TX FIFO.

amount of blocks: if your RFNoC image doesn't really have the need of several RFNoC blocks, you can set the number of blocks to be inserted in the FPGA by running the -m directive. In the example command, 5 blocks are being added to the .bit file.

Fill with FIFOs: You can list a number of blocks smaller than the maximum allowed (or than the number fixed with -m). In addition, you can fill the rest of free slots with FIFO blocks. In the example, one FIFO block is going to be added to the FPGA image.

Output file: the purpose of the whole 'make.py' script is to autogenerate an instantiation file and populate the source files needed for Vivado to build the FPGA satisfactorily. But sometimes you may only want to see the effect of adding your OOT module on the fpga/ directory, or how the instantiation file would look like. With the -o directive, you generate the instantiation file and save it at the desired path with the given name. In the example, it is saved at the tmp/ directory with the name verilog_CE.v. Keep in mind that if you don't give the -o directory, the rfnoc_ce_auto_inst_x3x0.v file will be overwritten and the FPGA image build process will start automatically.

The FPGA image building process may take over an hour.

NOTE: [Environment setup] - The script will also set up your Vivado environment by automatically running the 'setupenv.sh' located at /top/{device}. So you don't have to run this script before running the make.py. In addition, keep in mind that the setupenv.sh assumes that your Vivado is installed in the default location /opt/Xilinx/Vivado. If you chose a different installation directory, you may want to modify the setupenv.sh script.

Creating and running testbenches

In order to make the iteration process much quicker, we encourage you to design test benches for each of your RFNoC blocks before implementing them into the FPGA image. By doing so, you can detect bugs or flaws in a relatively early stage of your design. RFNoC modtool provides the structure for the testbenches for each of the out-of-tree blocks that you add with the "rfnocmodtool add" option.

You can see that in the Adding custom blocks section, the last files generated are:

  rfnoc/testbenches/noc_block_copy_tb folder created
  Adding file 'rfnoc/testbenches/noc_block_copy_tb/noc_block_copy_tb.sv'...
  Adding file 'rfnoc/testbenches/noc_block_copy_tb/Makefile'...
  Adding file 'rfnoc/testbenches/noc_block_copy_tb/CMakeLists.txt'...

The noc_block_copy_tb is a folder generated to contain all the files related to the test bench of the copy block. Each time that you create a new OOT block, a new folder will be generated as well. Inside of this folder, three files are located:

CMakeLists.txt: this is an empty file used, so far, only to increase the scope of the compilers.
noc_block_copy_tb.sv: this is a System Verilog file, in which your custom tests are to be located. This is the only file that you have to modify according with your design.
Makefile: This file determinates the directives that run the simulation.

This files require the Vivado tools to work, so you have to have this tool installed (preferably at the default location, /opt/Xilinx). Also, you would have to set up the Vivado environmental variables. Until now, you had to do this by hand, which is somewhat cumbersome. Now, with the use of the structure given by RFNoC modtool, the only thing that you have to do is build your own OOT module, which you have to do anyways! The builders will look up for the default locations of Vivado and the fpga repository and will set up your test benches so that they are ready to use, and you can fully focus on your design. To do so, simply create a build directory:

   mkdir build && cd build/

and then run cmake. If you are using PyBOMBS, cmake will detect automatically the location of the fpga repository. If you are installing by source, you'd have to provide the location where you installed it.

   cmake [-DUHD_FPGA_DIR=/PATH/TO/FPGA/REPOSITORY]

Then, and only for the first time, you have to run the following command:

   make test_tb

This command will modify your files by setting the correct path to the simulation tools. From now on, every time that you add a new block, this command will be run automatically.

Now you are ready to run the test bench! you don't have to go to the testbench directory, you can run it right there at the OOT main 'build/' directory! just run make noc_block_[name_of_your_block]_tb, which in this example case is:

   make noc_block_copy_tb

And the simulation will start. With every block that you create, also a make directive will be available for you to run the simulation right there at the main build directory.