NOT FINISHED DONT READ!

Overview

This application note gives a brief introduction into the steps required to start developing RFNoC blocks on your computer. First, you need to make sure that you have all the tools required to do RFNoC development. Then, we will set up a development sandbox in which you can do your RFNoC development. Finally, we will run through the basic steps required to actually write custom RFNoC blocks.

Prerequisites

For any of this to work, you need to make sure you have all the tools to do both software and FPGA development. Confirm you have the following tools installed:

Xilinx Vivado. The specific version depends on the branch and state of the FPGA code and is very important. Make sure to install the correct version.
Software build tools. If you can compile UHD from source, you have all the necessary components (If you're using PyBOMBS

Please make sure these tools are available before proceeding to the next step.

Creating a development environment

In the following, we will assume that GNU Radio is used as a helper tool for RFNoC development. Note that GNU Radio is by no means required to use or develop RFNoC, but it makes it a great deal easier. So in order to get your development environment set up, you will need the following software packages:

- UHD - GNU Radio - gr-ettus

The cleanest way to set this up is to install everything into a dedicated directory. PyBOMBS is the simplest way to do this. If you haven't already downloaded and installed PyBOMBS, you can do so by running the following commands:

   $ sudo pip install git+https://github.com/gnuradio/pybombs.git
   $ pybombs recipes add gr-recipes git+https://github.com/gnuradio/gr-recipes.git
   $ pybombs recipes add ettus git+https://github.com/EttusResearch/ettus-pybombs.git

From here, the easiest way to set up a PyBOMBS development environment is to run the following command:

   $ pybombs prefix init ~/prefix/rfnoc -R rfnoc [-a alias]

This will do the following:

- Create a directory in ~/prefix/rfnoc - Clone UHD, FPGA, GNU Radio, and gr-ettus sources into that directory - Compile and install all the software

Then, enable the prefix:

   $ cd ~/prefix/rfnoc
   $ source ./setup_env.sh

PyBOMBS is by no means required to do any of these steps. If you wish to install the software manually, follow the individual install notes for GNU Radio, UHD and gr-ettus and make sure they are reachable by linkers and compilers.

To manually download the software, use these git clone commands, which will select the correct branches:

   $ git clone --recursive -b rfnoc-radio-redo https://github.com/EttusResearch/uhd.git
   $ git clone -b maint https://github.com/gnuradio/gnuradio.git # master branch is also fine
   $ git clone -b radio-redo https://github.com/EttusResearch/gr-ettus.git

If you have already UHD, GNU Radio and/or gr-ettus installed, it would be sufficient to checkout to the branches mentioned and pull them. Thereafter, build each of the repositories (first UHD, then GNU Radio, then gr-ettus).

RFNoC Framework

Before explaining the tool utilization, it is better to have clarity on what is being done underneath your design. For this, lets review briefly how the RFNoC Framework structure is set up in our systems. The colourful image down here shows the basic structure of the RFNoC Stack.

From the stack there are clearly 5 different elements which a RFNoC developer has to take into account when creating an out-of-three block. How this elements interact with each other is explained in more detail at [2][3][4]. However, from here we could easily infer that there are, at least, 3 questions that our Out-of-three design had to answer until this day:

1. Does it have a complete FPGA Integration?: A verilog file that describes the hardware implementation is what allows us to take advantage of the powerful FPGA that is being shipped with the current and next gen USRPs. For the current stage of development, the .v file has to not only implement the custom desired functionality but also a efficient interaction with the AXI bus and other FPGA blocks. In addition, it the file has to be located in an specific location so that the building process can find it and use it as a resource for the IP generation.

2. Does it have a block declaration/control?: Now that the design has the hardware implementation in verilog, we would like to take the samples out of the USRP and receive them at our host. The block has to have a clear declaration which defines, at least, its inputs and outputs (which are self explanatory) and the NoC-ID, which is the method of identification between the HW part of our design and the SW part. Now, we might add some control over the samples that are being streamed into the block, and for that we have two options that depend on the complexity of the control that we want to apply: if as developers we can add control using NoC-Script, that could easily be added into the XML file. But if we need more exhaustive control and multiple data stream treatment, we might want to add more coding into it. Block controllers can be added as C++ in the UHD host, carefully generating the headers and adding them in the correct include folder and in the same manner, develop our control and add it in the library folder.

3. Does it have GNURadio integration?: GNURadio is where the application runs, and where more control over the host is applied, by adding block-specific settings, such as sample rate or bandwidths. In addition, is where we have the our block ready to connect it to other several blocks (which can be also RFNoC blocks). That means that we are almost done with our design. In order to do so, we have to add a new XML file that allows us to bind our design with GNURadio, and where we can edit the type on our I/Os, add variables for posterior signal processing, and a bunch of other design-specific parameters. In the same XML file we could add, just as we could have done at the block controllers, more block control by the means of NoC-Script. If we choose to do that, we could point our design to a "generic" implementation to control our GRC block. If we, otherwise, need even more control, we could choose to generate block code, that afterwards has to be located at gr-ettus: the headers as an include, the implementations as libraries.

So, you may think that this is just normal protocol of the RFNoC OOT block generation method. But, in order to answer this three questions with your OOT design, you have to take care of correct FPGA interfacing, correct API instantiation and right includes and correct location where to put your files... while you also put effort to the development of your custom design! Here at Ettus, we think that you should put all your effort in your custom design and let the RFNoC modtool do the dirty work.

RFNoC Modtool

In order to minimize the effort needed to implement a new RFNoC module, this gr_modtool-based tool provides the basic structure of the necessary files needed for the user to start with the development of his/her custom design right away. As it is gr_modtool-based, the manual found at [1] is also a great reference for this tool. Here, we discuss the features specific for the RFNoC modtool.

With the use of RFNoC ModTool, you can skip the struggle of locating the files of your design at different locations that depend (until now) on the location of your UHD and gr-ettus installation paths. Instead, you focus on your implementation and the tool takes care of boilerplate code and linking.

With the use of this tool, the questions stated in the last section would result in something like:

1. Does it have a complete FPGA Integration?: If not, start your design in the pregenerated verilog file, which also provides most of the standard AXI interface already done.

2. Does it have a block declaration/control?: Well, it has the files now, so write your functions if you need them. Don't even worry about modifying the files to add your NoC-ID: the tool will do that for you with the ID of your choice.

3 Does it have GNURadio integration?: It has it now. You just need to put additional control, in case you need it.

Don't worry about relocating your files, they are where they need to be: in your custom location. The tool generates the files so that the whole building process take into account your design regardless of the location of your OOT module in your machine.

Tool Installation

The tool is currently shipped within gr-ettus (radio-redo branch), so by installing gr-ettus, rfnocmodtool automatically becomes available.

Tool Utilization

After the installation, you are ready to use the tool to generate standard code for your Out-of-tree module. To check the usage of the tool, just type:

   $ rfnocmodtool help
   
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   Usage:
   rfnocmodtool <command> [options] -- Run <command> with the given options.
   rfnocmodtool help -- Show a list of commands.
   rfnocmodtool help <command> -- Shows the help for a given command. 
   
   List of possible commands:
   
   Name      Aliases          Description
   =====================================================================
   disable   dis              Disable block (comments out CMake entries for files) 
   info      getinfo,inf      Return information about a given module 
   remove    rm,del           Remove block (delete files and remove Makefile entries) 
   makexml   mx               Make XML file for GRC block bindings 
   add       insert           Add block to the out-of-tree module. 
   newmod    nm,create        Create a new out-of-tree module 
   rename    mv               Rename a block in the out-of-tree module.

If you have used the gr_modtool provided by GNURadio, you'll be familiar with this usage.

Creating an RFNoC OOT Module

At this point you are all set up to start generating your own RFNoC OOT and if you were thinking that it is just as easy as it is with gr_modtool, you are completely right! Simply go to your prefixes source location (~/prefix/rfnoc/src) and type:

   $ rfnocmodtool newmod [NAME OF THE MODULE]

In the following, assume we want to create a module with the name 'example'. You can write the name of your module right after the 'newmod' command, but if you happen to forget it, the tool will ask for it interactively. This will create a folder containing the basic folders that you may need for a functional module.

   $ rfnocmodtool newmod example
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   
   Creating out-of-tree module in ./rfnoc-example... Done.
   Use 'rfnocmodtool add' to add a new block to this currently empty module.
   
   $ ls rfnoc-example/
   $ apps  cmake  CMakeLists.txt  docs  examples  grc  include  lib  MANIFEST.md  python  README.md  rfnoc  swig

In contrast with gr_modtool, this includes a folder called 'rfnoc', which is where we put the UHD/FPGA files (more details below).

Adding custom blocks

Now you can start adding blocks to your module. The only thing to do is go inside of the module you just created and type:

   $ rfnocmodule add [NAME OF THE BLOCK]

For demonstrative purposes, lets create a block named 'copy'. As for the new module, the tool will ask you for the name of the block if you not put it in the first line. There is a bunch of directives that you could pass to the tool, but if you run it without any of these directives you should expect the following interactive parsing output:

   $ rfnocmodtool add copy
   linux; GNU C++ version 4.8.4; Boost_105400; UHD_003.010.rfnoc-radio-redo-0-fd4d734e
   
   RFNoC module name identified: example
   Block/code identifier: copy
   Enter valid argument list, including default arguments: 
   Add Python QA code? [Y/n] 
   Add C++ QA code? [y/N] 
   Block NoC ID (Hexadecimal): FFFF
   Skip Block Controllers Generation? [y/N] 
   Skip Block interface Generation? [y/N]

The tool is going to ask you about a valid argument list, in case you have that clear at this stage of your design, and for the generation of QA codes which are, such as in the case of gr_modtool, optional. Right after that, specific RFNoC questions appear:

NoC ID: This ID is a Hexadecimal number which serves as identification between the hardware part and the software part of the design. It can be as long as 16 0-9 A-F digits. If you dont provide the NoC ID, it will be set to a random number.
Block Controllers Generation: The block controllers are the C++ control that you can apply to the UHD-part of the design. In this files, you can add more control over this layer of your design. However, if you can manage to add the control you need over NoC-Script, you can omit the cpp/hpp block control files generation, reason why this is optional.
Block Interface: Analogously to the block controllers, you may want to add more design specific functionality to your design at the GNURadio interface. To do that, you add the logic into the block-interface files. In the same manner, if your control can be managed by adding NoC-Script to your xml file, you can disregard the generation of files.

Also, it is important to notice that if you don't intend to use the block controllers or you are not sure if you would, the presence of them in your design would do no harm. So, if in doubt, just add them. You may or may not add more functions inside them in a future stage of development of your OOT module. After finishing the parsing, the following files are going to be generated/edited:

   Adding file 'lib/copy_impl.h'...
   Adding file 'lib/copy_impl.cc'...
   Adding file 'include/example/copy.h'...
   Adding file 'include/example/copy_block_ctrl.hpp'...
   Adding file 'lib/copy_block_ctrl_impl.cpp'...
   Editing swig/example_swig.i...
   Adding file 'python/qa_copy.py'...
   Editing python/CMakeLists.txt...
   Adding file 'grc/example_copy.xml'...
   Adding file 'rfnoc/blocks/copy.xml'...
   Adding file 'rfnoc/fpga-src/noc_block_copy.v'...

And, at this stage, you are all set up to start adding your custom functionalities.

Building the FPGA image

After you are done with your block development, you may want to generate the fpga image to actually run your implementation. A small script to help you do this is provided within the fpga repository that you installed in the Creating a development environment section.

The script is called 'make', and it is located at {INSTALL_PREFIX}/fpga/usrp3/tools/scripts. You can run the help by typing:

   make.py --help
   
   Generate the NoC block instantiation file
   
   positional arguments:
     blocks                List block names to instantiate.
   
   optional arguments:
     -h, --help            show this help message and exit
     -I INCLUDE_DIR, --include-dir INCLUDE_DIR
                           Path directory of the RFNoC Out-of-Tree module fpga srcs
                           {OOT_moddir}/rfnoc/fpga-src
     -m MAX_NUM_BLOCKS, --max-num-blocks MAX_NUM_BLOCKS
                           Maximum number of blocks (Max. Allowed for x310|x300:
                           11, for e300:6)
     --fill-with-fifos     If the number of blocks provided was smaller than the
                           max number, fill the rest with FIFOs
     -o OUTFILE, --outfile OUTFILE
                           Output /path/filename - By running this directive, you
                           won't build your IP
     -d DEVICE, --device DEVICE
                           Device to be programmed [x300, x310, e310]
     -t TARGET, --target TARGET
                           Build target - image type [X3X0_RFNOC_HG,
                           X3X0_RFNOC_XG, E310_RFNOC_sg3...]

As it can be seen, there is also a good amount of arguments that you can pass to this script. lets analyse an example of this: suppose that the OOT module that we created on the [last step] is ready to be burned into the fpga, and for that we are going to use this command:

   make.py fft copy fft window -I {OOT_moddir/rfnoc-example}/rfnoc/fpga-src/ -d x310 -t X310_RFNOC_HG -m 5 --fill-with-fifos -o ~/tmp/verilog_CE.v

and this command already tells us a lot of insights on how to use this script.

Blocks: right after the script name you list the blocks that you want to have in your image separated by a space. They can be from the blocks that you have from your OOT module or from the ones that are provided from Ettus, or a combination of those as well. There are two things to keep in mind while listing blocks here:
- So far, there is a restriction on the amount of blocks that you can add into the FPGA image: 11 for the X3X0 devices, and 6 for the E3X0.
- The FPGAs provided in the USRPs are powerful and have rather high capacity. However, if the overall logic connections needed to implement the blocks that you listed is rather huge, then your FPGA may run out space and your synthesis may fail. For the example, we list two fft blocks and one window block (which are blocks provided with the installation) and one copy block (which is part of our custom design).

Although the amount of blocks is constantly increasing (thanks to the constant development and contribution of the community), here is a brief overview of the blocks from which you can choose:

LIST

Include_dir: Sure, you can add your custom block to the FPGA Image, you just have to tell the Vivado builder where to find it. Here, with the -I directive you give the /Path/to/your/fpga-src, which is inside the rfnoc/ folder of your RFNoC OOT module. It is important to notice that inside this folder there is also a file called 'Makefile.srcs', in which you can find the path of your OOT and a list of all the OOT blocks that you have developed in your custom module. This is an autogenerated file, which is populated everytime that you add a block. Manually modifying this block may result in inconveniences.

Device: nowadays the USRP3 devices support RFNoC, but the nextGen USRPs will support RFNoC as well. Having this in mind, with this -d directive you tell the script in which device you intend to use the image. By omitting this directive, the script will choose the X310 device as default. In the example, however, we chose explicitly the -d x310

Target: there are several build options, and you can choose the one that suits better with your design with the -t directive. Detailed information about the build targets can be found at the build instructions for USRP3, found under [foo]. If not -t is given, a default target will be chosen for the given device. In the example, the -t X310_RFNOC_HG builds our design intended for the USRP X310 device, with 1GigE on SFP+ Port0, 10Gig on SFP+ Port1. DRAM TX FIFO.

amount of blocks: if your RFNoC image doesn't really have the need of several RFNoC blocks, you can set the number of blocks to be inserted in the FPGA by running the -m directive. In the example command, 5 blocks are being added to the .bit.

Fill with FIFOs: You can list a number of blocks smaller than the maximum allowed (or than the number fixed with -m). In addition, you can fill the rest of free slots with FIFO blocks. In the example, one FIFO block is going to be added to the FPGA image.

Output file: the purpose of the whole 'make.py' script is to autogenerate an instantiation file and populate the source files needed for Vivado to build the FPGA satisfactorily. But sometimes you may only want to see the effect of adding your OOT module on the fpga/ directory, or how the instantiation file would look like. With the -o directive, you generate the instantiation file and save it at the desired path with the given name. In the example, it is saved at the tmp/ directory with the name verilog_CE.v. Keep in mind that if you don't give the -o directory, the rfnoc_ce_auto_inst_x3x0.v file will be overwritten and the FPGA image build process will start automatically.

NOTE: [Environment setup] - The script will also set up your Vivado environment by automatically running the 'setupenv.sh' located at /top/{device}. So you don't have to run this script before running the make.py. In addition, keep in mind that the setupenv.sh assumes that your Vivado is installed in the default location /opt/Xilinx/Vivado. If you chose a different installation directory, you may want to modify the setupenv.sh script.

The FPGA image building process takes several minutes.