Table of Contents
{m-note m-warning}
- Installation Warning
- If you plan to use the included plotting from the cpp code, you will need to make sure that you have matplotlib and python 2.7 installed. We use the to matplotlib-cpp to call this external library and generate the desired figures. Please see Additional Evaluation Requirements for more details on the exact install.
Collection
The first step in any evaluation is to first collect the estimated trajectory of the proposed systems. Since we are interested in robotic application of our estimators we want to record the estimate at the current timestep (as compared to a "smoothed" output or one that includes loop-closures from future timesteps). Within the ROS framework, this means that we just need to publish the current estimate at the current timestep. We recommend using the following ov_eval::Recorder utility for recording the estimator output directly into a text file. Works with topics of type PoseWithCovarianceStamped, PoseStamped, TransformStamped, and Odometry.
Transformation
We now need to ensure both our estimated trajectory and groundtruth are in the correct formats for us to read in. We need things to be in the RPE text file format (i.e. time(s), px, py, pz, qx, qy, qz, qw). We have a nice helper script that will transform ASL / EuRoC groundtruth files to the correct format. By default the EuRoC groundtruth has the timestamp in nanoseconds and the quaternion is in an incorrect order (i.e. time(ns), px, py, pz, qw, qx, qy, qz). A user can either process all CSV files in a given folder, or just a specific one.
In addition we have a specific folder structure that is assumed. We store trajectories by first their algorithm name and then a folder for each dataset this algorithm was run on. The folder names of the datasets need to match the groundtruth trajectory files which should be in their own separate folder. Please see the example recorded datasets for how to structure your folders.
Processing & Plotting
Now that we have our data recorded and in the correct format we can now work on processing and plotting it. In the next few sections we detail how to do this for absolute trajectory error, relative pose error, normalized estimation error squared, and bounded root mean squared error plots. We will first process the data into a set of output text files which a user can then use to plot the results in their program or language of choice. The align mode of all the following commands can be of type posyaw, posyawsingle, se3, se3single, sim3, and none.
Script "plot_trajectories"
To plot the data we can us the following command which will plot a 2d xy and z-time position plots. It will use the filename as the name in the legend, so you can change that to change the legend or edit the code.
Script "error_singlerun"
The single run script will plot statistics and also 3 
bounds if available. One can use this to see consistency of the estimator or debug how the current run has gone. It also reports to console the average RMSE and RPE values for this run along with the number of samples. To change the RPE distances you will need to edit the code currently.
Example output:
Script "error_dataset"
This dataset script will evaluate how a series of algorithms compare on a single dataset. Normally this is used if you just have single dataset you want to compare algorithms on, or compare a bunch variations of your algorithm to a simulated trajectory. In the console it will output the ATE 3D and 2D, along with the 3D RPE and 3D NEES for each method after it performs alignment. To change the RPE distances you will need to edit the code currently.
Example output:
Script "error_comparison"
This compares all methods to each other on a series of datasets. For example, you run a bunch of methods on all the EurocMav datasets and then want to compare. This will do the RPE over all trajectories, and an ATE for each dataset. It will print the ATE and RPE for each method on each dataset in the console. Then following the Filter Evaluation Metrics, these are averaged over all the runs and datasets. Finally at the end it outputs a nice latex table which can be directly used in a paper. To change the RPE distances you will need to edit the code currently.
Example output:
