This is the user manual for the White Rabbit ptp Core developed on
ohwr.org. It describes the building and running process. If you don't
want to get your hands dirty and prefer to use the binaries available at
http://www.ohwr.org/projects/wr-cores/files please skip
Building the Core and move forward directly to
Running and Configuring.
This manual is about the official wr-nic-v1.0-241-g65acb99 stable release of White Rabbit PTP Core (WRPC).
The code and documentation for the project is distributed in the following places:
http://www.ohwr.org/projects/wr-cores/documentshttp://www.ohwr.org/projects/wr-cores/filesgit://ohwr.org/hdl-core-lib/wr-cores.gitgit://ohwr.org/hdl-core-lib/wr-cores/wrpc-sw.gitOther tools useful for building and running wrpc can be downloaded from the following locations:
http://www.ohwr.org/projects/hdl-make/fileshttp://www.ohwr.org/attachments/download/1133/lm32.tar.xzwrpc-sw project.
The repositories containing the wrpc gateware and software (wr-cores,
wrpc-sw) are tagged with wrpc-v2.1 tag. Other tools
used to build the core and load it into spec board should be used in their
newest stable release.
Any official hot fixes, if any, for this release will live in the branch called
wrpc-v2.1-fixes, in each wrpc repository.
The absolutely minimum hardware you need to build and run the White Rabbit ptp Core is a PC computer with Linux and one Simple PCIe fmc Carrier (spec) - http://www.ohwr.org/projects/spec. However, it is highly recommended to use also the dio fmc card (http://www.ohwr.org/projects/fmc-dio-5chttla) for storing calibration values and configuration in eeprom(described later) as well as feeding 1-PPS and 10MHz from external clock and outputting 1-PPS aligned to WR time. To test the White Rabbit synchronization, you will also need:
Note: you can skip this chapter if you want to use the release binaries available for download from ohwr.org.
Building the White Rabbit ptp Core is a two step process. First you have to synthesize the fpga firmware (gateware) and then compile the software which will be running on a softcore lm32 CPU instantiated inside the gateware. Optionally, you can skip the compilation of lm32 software, since the synthesized gateware contains default lm32 software.
To perform the steps below you will need a computer running Linux.
Before running the synthesis process you have to make sure that your environment is set up correctly. You need the Xilinx ISE software with at least free of charge WebPack license. ISE provides a set of scripts: settings32.sh, settings32.csh, settings64.sh and settings64.csh that configure all the system variables required by the Xilinx software. Depending on the shell you use and whether your Linux is 32 or 64-bits you should execute one of them before other tools are used. For 32-bit system and BASH shell you should call:
/opt/Xilinx/<version>/ISE_DS/settings32.sh
The easiest way to ensure that ISE-related variables are always set in your shell is adding the execution of the script above to your bash.rc file. You can check if the shell is configured correctly by verifying if $XILINX variable contains the path to your ISE installation directory.
Note: current version of hdlmake tool developed at CERN requires modification of $XILINX variable after settings script execution. This (provided that the installation path for ISE is /opt/Xilinx/<version>) should be the following:
$ export XILINX=/opt/Xilinx/<version>/ISE_DS
Note: the Xilinx project file included in the wrpc sources was created with Xilinx ISE 14.5. It is however recommended to use the newest available version of ISE software.
HDL sources for wr ptp Core could be synthesized using Xilinx ISE without any additional tools, but using hdlmake is more convenient. It creates a synthesis Makefile and ISE project file based on a set of Manifest.py files deployed among the directories inside the wr-cores repository.
First, please download the hdlmake binary from its location given in Repositories and Releases and make it executable. At the time this document is written, the most recent stable version of hdlmake is 1.0:
$ wget http://www.ohwr.org/attachments/download/2070/hdlmake-v1.0
$ chmod a+x hdlmake-v1.0
It is recommended to add the hdlmake binary location to your $PATH environment variable to be able to call it from any directory:
$ export PATH=<your_hdlmake_location>:$PATH
Note: the hdlmake usage instructions here are based on version 1.0. If you use more recent release or a development version, please be aware that its execution parameters may change. In that case please refer to hdlmake documentation.
Having Xilinx ISE software and hdlmake in place, you can clone the main wr ptp core git repository and start building the fpga bitstream. First, please create a local copy of the wr-cores in the preferred location in your system.
$ git clone git://ohwr.org/hdl-core-lib/wr-cores.git <your_location>/wr-cores
$ cd <your_location>/wr-cores
To build the gateware using sources of a stable release wrpc-v2.1, you have to checkout the proper git tag:
$ git checkout wrpc-v2.1
If you use wr-cores within another project (like wr-nic), you may need to check out another release tag for this repository. Please refer to the project's documentation to find out which version of this package you need to build.
You also need to fetch other git repositories containing modules essential for wr ptp core. They are configured as git submodules inside the wr-cores repository:
$ git submodule init
$ git submodule update
The local copies of the submodules are stored to:
<your_location>/wr-cores/ip_cores
The subdirectory which contains the main synthesis Manifest.py for spec board and in which you should perform the whole process is:
$ cd <your_location>/wr-cores/syn/spec_1_1/wr_core_demo/
First, please call hdlmake to create synthesis Makefile for Xilinx ISE:
$ hdlmake-v1.0 --make-ise --ise-proj
After that, the actual synthesis is just the matter of executing:
$ make
This takes (depending on your computer speed) about 15 minutes and should create two files with fpga firmware: spec_top.bit and spec_top.bin. The former can be downloaded to fpga with Xilinx Platform Cable using e.g. Xilinx Impact. The latter can be used with the kernel drivers from spec-sw repository (check example in Running and Configuring).
If, on the other hand, you would like to clean-up the repository and rebuild everything from scratch you can use the following commands:
Note: By default, the release lm32 software is embedded inside the FPGA bitstream you've downloaded from ohwr.org or synthesized in the previous chapter. That means you don't have to do manual compilation of lm32 software unless you want to use a development version or you've made some changes required by your application.
To compile the lm32 software for White Rabbit ptp Core you will need to download and unpack the lm32 toolchain from the location mentioned in Repositories and Releases:
$ wget http://www.ohwr.org/attachments/download/1133/lm32.tar.xz
$ tar xJf lm32.tar.xz -C <your_lm32_location>
Then you need to set the CROSS_COMPILE variable in order to compile the software for a lm32 processor:
$ export CROSS_COMPILE="<your_lm32_location>/lm32/bin/lm32-elf-"
To get the sources of wrpc software please clone the wrpc-sw git repository tagged with wrpc-v2.1 tag. If you use wrpc within another project, you may need to checkout a different tag or a specific commit; if this applies, please refer to the documentation of the other package to find the exact version you need to reproduce the released binaries before you make your changes.
$ git clone git://ohwr.org/hdl-core-lib/wr-cores/wrpc-sw.git <your_location>/wrpc-sw
$ cd <your_location>/wrpc-sw
$ git checkout wrpc-v2.1 # or "git checkout master"
Before you can compile wrpc-sw you need to make a few configuration choices. The package uses Kconfig as a configuration engine, so you may run one of the following commnads (the first is text-mode, the second uses a KDE GUI and the third uses a Gnome GUI):
$ make menuconfig
$ make xconfig
$ make gconfig
Other Kconfig target applies, like config, oldconfig
and so on. A few default known-good configurations are found in
./configs and you choose one by makeing it by name:
$ make spec_defconfig
The most important configuration choice at this point in time is
whether to enable Etherbone or not. It is disabled by default in
spec_defconfig and enabled by default in
etherbone_defconfig.
After the package is configured, just run make without
parameters to build your binary file:
$ make
The first time you build, the Makefile automatically downloads the git submodules of this package, unless you already did that by hand. The second and later build won't download anything from the network.
The resulting binary wrc.bin can be then used with the loader from spec-sw software package to program the lm32 inside the White Rabbit ptp Core (Running and Configuring).
There is a spec board software support(spec-sw) project in ohwr.org. It contains a set of Linux kernel drivers and user space tools, written by Alessandro Rubini and Tomasz Wlostowski, that interact with a spec board plugged into a PCI-Express slot.
Instructions in this section are based on the release 2013-05 of spec-sw and are limited to absolute minimum required to load wrpc fpga and lm32 firmware. Full manual for spec-sw can be found at:
http://www.ohwr.org/attachments/download/2134/spec-sw-2013-05-release.pdf
If there is a more recent version of the spec software support, the up-to-date documentation can always be found in doc/ subdirectory of spec-sw git repository.
First, please clone the git repository of spec-sw package and build it:
$ git clone git://ohwr.org/fmc-projects/spec/spec-sw.git <your_specsw_location>
$ cd <your_specsw_location>
$ git checkout spec-sw-v2013-05
$ make
Then you have to copy your spec_top.bin generated in HDL synthesis or downloaded from ohwr.org to /lib/firmware/fmc/. changing its name:
Note: the commands below have to be executed with superuser rights
$ sudo cp <your_location>/wr-cores/syn/spec_1_1/wr_core_demo/spec_top.bin \
/lib/firmware/fmc/spec-demo.bin
You have to download also the "golden" firmware for spec card. It is used by the drivers to recognize the hardware:
$ wget http://www.ohwr.org/attachments/download/1756/spec-init.bin-2012-12-14
$ sudo mv spec-init.bin-2012-12-14 /lib/firmware/fmc/spec-init.bin
Now, you are ready to load necessary drivers that configure the Spartan 6 fpga with the given bitstream (make sure you are in <your_specsw_location>):
$ sudo insmod fmc-bus/kernel/fmc.ko
$ sudo insmod kernel/spec.ko
$ sudo insmod fmc-bus/kernel/fmc-trivial.ko gateware=fmc/spec-demo.bin
To check if the fpga firmware file was found by the driver and correctly loaded to fpga the dmesg Linux command can be called. Among plenty of messages you should be able to find something very similar to:
[1639675.431979] spec 0000:0b:00.0: probe for device 000b:0000
[1639675.431992] spec 0000:0b:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[1639675.435246] spec 0000:0b:00.0: got file "fmc/spec-init.bin", 1484404 (0x16a674) bytes
[1639675.625773] spec 0000:0b:00.0: FPGA programming successful
[1639675.994110] spec 0000:0b:00.0: mezzanine 0
[1639675.994111] EEPROM has no FRU information
[1639705.910703] fmc fmc-0b00: Driver has no ID: matches all
[1639705.910731] spec 0000:0b:00.0: reprogramming with fmc/spec-demo.bin
[1639706.104417] spec 0000:0b:00.0: FPGA programming successful
If everything went right up to this moment you have your board running the fpga bitstream with default lm32 software. If you want to load your own wrc.bin built from wrpc-sw repository you can use the spec-cl tool. Programming is done with the simple command below:
$ sudo tools/spec-cl <your_location>/wrpc-sw/wrc.bin
Now you should be able to start the Virtual-UART tool (also part of the spec-sw package) that will be used to interact with the White Rabbit ptp Core Shell:
$ sudo tools/spec-vuart
If you are able to see the wrpc Shell prompt wrc# that means the Core is up and running on your spec. Congratulations !
By default wrpc starts in wr Slave mode and uses the default calibration values for Axcen AXGE-3454-0531 sfp. This might be fine for running wrpc for the first time and synchronizing it to another spec running the same firmware. It is however recommended to perform few additional steps through the wrpc shell.
Note: the examples below describe only a subset of wrpc Shell commands required to make a basic configuration and calibration. A full description of all supported commands can be found in WRPC Shell commands.
Before making the configuration changes, it is good to stop the ptp daemon. Then, the debug messages from daemon would not show up to the console while you will interact with the shell.
wrc# ptp stop
If your spec has any Mezzanine board plugged into the fmc connector (e.g. dio, Fine Delay, tdc...) then you can create a calibration database inside the fmc eeprom. The example below presents wrpc Shell commands which create an empty sfp database and add two Axcen transceivers with deltaTx, deltaRx and alpha parameters associated with them.
wrc# sfp erase
wrc# sfp add AXGE-1254-0531 180456 148066 72169888
wrc# sfp add AXGE-3454-0531 180456 148066 -73685416
To check the content of the sfp database you can execute the sfp show shell command.
Note: The deltaTx and deltaRx parameters above are the default ones for wrpc-2.1 release bitstream available on ohwr.org and most probably will be the cause of some constant offset when wrpc is re-synthesized or connected to the WR Switch. To find the new values you should read the wr Calibration procedure (http://www.ohwr.org/documents/213).
The wr ptp Core's mode of operation (GrandMaster/Master/Slave) can be set using the mode shell command:
wrc# mode gm # for GrandMaster mode
wrc# mode master # for Master mode
wrc# mode slave # for Slave mode
This stops the ptp daemon, changes the mode of operation, but does not start it back automatically. Therefore after changing it you need to start the daemon manually:
wrc# ptp start
Note: For running in GrandMaster mode, you need to provide 1-PPS and 10MHz signal from external source (e.g. GPS receiver or Cesium clock). Please connect then 1-PPS signal to LEMO No.4 and 10MHz to LEMO No.5 connector of fmc dio board.
One option is to type all those commands to initialize the wrpc software to the required state every time the Core starts. However, you can also write your own init script to the eeprom and wrpc software will execute it each time it starts (also when coming back from reset after programming the lm32). A simple script that reads sfp parameters from the eeprom, configures the WR mode to Slave and starts the ptp daemon is presented below:
wrc# init erase
wrc# init add ptp stop
wrc# init add sfp detect
wrc# init add sfp match
wrc# init add mode slave
wrc# init add calibration
wrc# init add ptp start
Almost exactly the same one can be used for running wrpc in GrandMaster or Master mode. The only difference would be init add mode slave vs. init add mode gm or init add mode master.
Having the sfp database, and the init script created in Writing EEPROM and calibration you can restart the wr ptp Core by reprogramming the lm32 software (with spec-cl tool) or by typing the shell command:
wrc# init boot
You should see the log messages that confirm the init script execution:
(...)
WR Core: starting up...
W1: f80000036c38ee28
get_persistent_mac: Using W1 serial number
ID: cafebabe
t24p read from EEPROM: 6900 ps
Loops per jiffy: 20820
Locking PLL
executing: ptp stop
executing: sfp detect
AXGE-1254-0531
executing: sfp match
SFP matched, dTx=180585, dRx=145855, alpha=72169888
executing: mode slave
Locking PLL
executing: calibration
Found phase transition in EEPROM: 6900ps
executing: ptp start
Now you should have the White Rabbit ptp Core running in wr Slave mode. wrpc needs to make a calibration of t24p phase transition value. It has to be done only once for a new bitstream and is performed automatically when wrpc runs in Slave mode. That is why it is very important, even if wrpc is meant to run in Master mode, to configure it to Slave for a moment and connect it to any wr Master. That has to be repeated every time a new bitstream (gateware) is deployed. Measured value is automatically stored to EEPROM and used later in Master or GrandMaster mode.
The Shell also contains the monitoring function which you can use to check the wr synchronization status:
wrc# gui
The information is presented in a clear, auto-refreshing screen. To exit from this console mode press <Esc>. Full description about information reported by gui is provided in WRPC GUI elements.
Note: the Synchronization status and Timing parameters in gui are available only in wr Slave mode. When running as wr Master, you would be able to see only the current date and time, link status, Tx and Rx packet counters, lock and calibration status.
If you want to store the statistics from wrpc operation, it's probably better to use the stat cont shell command. It reports the same information as GUI but in a form which is easier to parse and analyze:
wrc# stat cont
lnk:1 rx:416 tx:118 lock:1 sv:1 ss:'TRACK_PHASE' aux:0 sec:94197 \
nsec:793068184 mu:836241 dms:400556 dtxm:10 drxm:163610 dtxs:0 drxs:128400 \
asym:35129 crtt:544221 cko:-5 setp:7667 hd:61479 md:37221 ad:65000 ucnt:101 \
temp: 45.6875 C
lnk:1 rx:417 tx:119 lock:1 sv:1 ss:'TRACK_PHASE' aux:0 sec:94198 \
nsec:293076296 mu:836253 dms:400562 dtxm:10 drxm:163610 dtxs:0 drxs:128400 \
asym:35129 crtt:544233 cko:-4 setp:7663 hd:61485 md:37259 ad:65000 ucnt:102 \
temp: 45.6875 C
(...)
If you have a dio Mezzanine board plugged to your spec, you can check the synchronization performance by observing the offset between 1-PPS signals from the wr Master and wr Slave. White Rabbit ptp Core generates 1-PPS signal to the LEMO connector No. 1 on dio Mezzanine. However, please remember to use oscilloscope cables having the same length and type (with the same delay), or take their delay difference into account in your measurements.
My computer hangs on loading spec.ko or fmc-trivial.ko driver.
This will occur when you try to load the spec.ko kernel driver while your spec-vuart is running and trying to get messages from Virtual-UART's registers inside wrpc. Please remember to quit spec-vuart before reloading the driver.
I want to synthesize wrpc but hdlmake does nothing, just quits without any message.
Please check if you have the Xilinx ISE-related system variables set correctly (settings32.sh script provided by Xilinx sets them) and make sure you have overwritten the $XILINX variable to:
$ export XILINX=/opt/Xilinx/<version>/ISE_DS
or similar, if your installation folder differs from default.
wr ptp Core seems to work but I observe on my oscilloscope that the offset between 1-PPS signals from wr Master and wr Slave is more than 1 ns.
If you're trying to synchronize the spec board to wr Switch please remember to read the document and perform the wr Calibration to find out the values of deltaRx and deltaTx for your firmware. Check if the oscilloscope cables you use have the same delays (or take the delay difference into account in your measurements).
If you have found a bug, you have problems with White Rabbit ptp Core or one
of the tools used to build and run it, you can write to our mailing list
white-rabbit-dev@ohwr.org
help reports the available commands in this instance of wrpc
| |
ver | prints which version of wrpc is running
|
config | prints the Kconfig file used to build this instance of wrpc. It is an optional command, enabled at build time by CONFIG_CMD_CONFIG
|
verbose <digits> | Set PPSi verbosity. See the PPSi manual about the meaning of the digits (hint: verbose 1111 is a good first bet too see how the ptp system is working)
|
pll init <mode> <ref_channel> <align_pps> | manually run spll_init()
function to initialize SoftPll
|
pll cl <channel> | check if SoftPLL is locked for the channel
|
pll sps <channel> <picoseconds> | set phase shift for the channel
|
pll gps <channel> | get current and target phase shift for the channel
|
pll start <channel> | start SoftPLL for the channel
|
pll stop <channel> | stop SoftPLL for the channel
|
pll sdac <index> <val> | set the dac
|
pll gdac <index> | get dac's value
|
gui | starts GUI wrpc monitor
|
stat | prints one line log message
|
stat cont | prints log message for each second (Esc to exit back to shell)
|
stat bts | prints bitslide value for established wr Link, needed by calibration procedure
|
ptp start | start wr ptp daemon
|
ptp stop | stops wr ptp daemon
|
mode | prints available wr ptp modes
|
mode gm|master|slave | sets wrpc to operate as Grandmaster clock (requires external 10MHz and 1-PPS reference), ptp Master or ptp Slave. After setting the mode ptp start must be re-issued
|
calibration | tries to read t2/4 phase transition value from eeprom (in WR Master or GrandMaster mode), or executes the t24p calibration procedure and stores its result to EEPROM (in WR Slave mode)
|
time | prints current time from wrpc
|
time raw | prints current time in a raw format (seconds, nanoseconds)
|
time set <sec> <nsec> | sets wrpc time
|
time setsec <sec> | sets only seconds part of wrpc time (useful for setting time in GrandMaster mode, when nanoseconds counter is aligned to external 1-PPS and 10 MHz)
|
time setnsec <nsec> | sets only nanoseconds part of wrpc time
|
sfp detect | prints the ID of currently used sfp transceiver
|
sfp erase | cleans the sfp database stored in fmc eeprom
|
sfp add <ID> <deltaTx> <deltaRx> <alpha> | stores calibration parameters for sfp to the database in fmc eeprom
|
sfp show | prints all sfp transceivers stored in database
|
sfp match | tries to get calibration parameters from database for currently used sfp transceiver (sfp detect must be executed before match)
|
init erase | cleans initialization script in fmc eeprom
|
init add <cmd> | adds shell command at the end of initialization script
|
init show | prints all commands from the script stored in eeprom
|
init boot | executes the script stored in fmc eeprom (the same action is done automatically when wrpc starts after resetting lm32)
|
mac get | prints wrpc's mac address
|
mac getp | re-generates mac address from 1-wire digital thermometer or eeprom
|
mac set <mac> | sets the mac address of wrpc
|
mac setp <mac> | sets mac address to the 1-wire eeprom (if available)
|
sdb | prints devices connected to the Wishbone bus inside wrpc
|
ip get
| |
ip set <ip> | reports or sets the IPv4 address of the wrpc (only available if CONFIG_ETHERBONE is set at build time
|
w1w <offset> <byte> [<byte> ...]
| |
w1r <offset> <len> | If CONFIG_W1 is set and a OneWire eeprom esists, write and read data. For writing, byte values are decimal
|
TAI Time: | current state of device's local clock
|
RX: / TX: | Rx/Tx packets counters
|
mode: | operation mode of White Rabbit ptp Core - <WR
Master, WR Slave>
|
< Locked, NoLock > | SoftPLL lock state
|
Servo state: | current state of wr servo state machine -
<Uninitialized, SYNC_SEC, SYNC_NSEC, SYNC_PHASE, TRACK_PHASE>
|
Phase tracking: | is phase tracking enabled when wr Slave is
synchronized to wr Master - <ON, OFF>
|
Synchronization source: | network interface name from which wr
daemon gets synchronization - <wru1>
|
Round-trip time (mu): | round-trip delay in picoseconds
(delay_MM)
|
Master-slave delay: | estimated one-way (master to slave) link
delay (delay_MS)
|
Master PHY delays: | transmission/reception delays of wr
Master's hardware (\Delta_TXM, \Delta_RXM)
|
Slave PHY delays: | transmission/reception delays of wr Slave's
hardware (\Delta_TXS, \Delta_RXS)
|
Total link asymmetry: | wr link asymmetry calculated as
delay_MM - 2 \cdot delay_MS
|
Cable rtt delay: | round-trip fiber latency
|
Clock offset: | Slave to Master offset calculated by ptp daemon
(offset_MS)
|
Phase setpoint: | current Slave's clock phase shift value
|
Skew: | the difference between current and previous estimated
one-way link delay
|
Update counter: | the state of counter incremented every time
the wr servo is updated
|