Class Benchmark

Nested Relationships

Nested Types

Class Documentation

class Benchmark

Benchmark a set of planners on a problem instance.

Public Types

using RunProgressData = std::vector<std::map<std::string, std::string>>
using PreSetupEvent = std::function<void(const base::PlannerPtr&)>

Signature of function that can be called before a planner execution is started.

using PostSetupEvent = std::function<void(const base::PlannerPtr&, RunProperties&)>

Signature of function that can be called after a planner execution is completed.

Public Functions

Benchmark(geometric::SimpleSetup &setup, const std::string &name = std::string())

Constructor needs the SimpleSetup instance needed for planning. Optionally, the experiment name (name) can be specified.

Benchmark(control::SimpleSetup &setup, const std::string &name = std::string())

Constructor needs the SimpleSetup instance needed for planning. Optionally, the experiment name (name) can be specified.

virtual ~Benchmark() = default
void addExperimentParameter(const std::string &name, const std::string &type, const std::string &value)

Add an optional parameter’s information to the benchmark output. Useful for aggregating results over different benchmark instances, e.g., parameter sweep. type is typically “BOOLEAN”, “INTEGER”, or “REAL”.

const std::map<std::string, std::string> &getExperimentParameters() const

Get all optional benchmark parameters. The map key is ‘name type’

std::size_t numExperimentParameters() const

Return the number of optional benchmark parameters.

void setExperimentName(const std::string &name)

Set the name of the experiment.

const std::string &getExperimentName() const

Get the name of the experiment.

void addPlanner(const base::PlannerPtr &planner)

Add a planner to use.

void addPlannerAllocator(const base::PlannerAllocator &pa)

Add a planner allocator to use.

void clearPlanners()

Clear the set of planners to be benchmarked.

void setPlannerSwitchEvent(const PreSetupEvent &event)

Set the event to be called before any runs of a particular planner (when the planner is switched)

void setPreRunEvent(const PreSetupEvent &event)

Set the event to be called before the run of a planner.

void setPostRunEvent(const PostSetupEvent &event)

Set the event to be called after the run of a planner.

virtual void benchmark(const Request &req)

Benchmark the added planners on the defined problem. Repeated calls clear previously gathered data.

Note

The values returned for memory consumption may be misleading. Memory allocators often free memory in a lazy fashion, so the returned values for memory consumption indicate the increase in memory usage for each run. Since not all the memory for the previous run was freed, the increase in usage may be close to 0. To get correct averages for memory usage, use req.runCount = 1 and run the process multiple times.

Parameters:

req – The parameters for the execution of the benchmark

const Status &getStatus() const

Get the status of the benchmarking code. This function can be called in a separate thread to check how much progress has been made.

const CompleteExperiment &getRecordedExperimentData() const

Return all the experiment data that would be written to the results file. The data should not be changed, but it could be useful to quickly extract cartain statistics.

virtual bool saveResultsToStream(std::ostream &out = std::cout) const

Save the results of the benchmark to a stream.

bool saveResultsToFile(const char *filename) const

Save the results of the benchmark to a file.

bool saveResultsToFile() const

Save the results of the benchmark to a file. The name of the file is the current date and time.

Protected Attributes

geometric::SimpleSetup *gsetup_

The instance of the problem to benchmark (if geometric planning)

control::SimpleSetup *csetup_

The instance of the problem to benchmark (if planning with controls)

std::vector<base::PlannerPtr> planners_

The set of planners to be tested.

CompleteExperiment exp_

The collected experimental data (for all planners)

Status status_

The current status of this benchmarking instance.

PreSetupEvent plannerSwitch_

Event to be called when the evaluated planner is switched.

PreSetupEvent preRun_

Event to be called before the run of a planner.

PostSetupEvent postRun_

Event to be called after the run of a planner.

struct CompleteExperiment

This structure holds experimental data for a set of planners.

Public Members

std::string name

The name of the experiment.

std::vector<PlannerExperiment> planners

The collected experimental data; each element of the array (an experiment) corresponds to a planner.

double maxTime

The maximum allowed time for planner computation during the experiment (seconds)

double maxMem

The maximum allowed memory for planner computation during the experiment (MB)

unsigned int runCount

The number of runs to execute for each planner.

time::point startTime

The point in time when the experiment was started.

double totalDuration

The amount of time spent to collect the information in this structure (seconds)

std::string setupInfo

The output of SimpleSetup::print() before the experiment was started.

std::uint_fast32_t seed

The random seed that was used at the start of the benchmark program.

std::string host

Hostname that identifies the machine the benchmark ran on.

std::string cpuInfo

Information about the CPU of the machine the benchmark ran on.

std::map<std::string, std::string> parameters

Additional, experiment specific parameters. This is optional.

struct PlannerExperiment

The data collected after running a planner multiple times.

Public Functions

inline bool operator==(const PlannerExperiment &p) const

Public Members

std::string name

The name of the planner.

std::vector<RunProperties> runs

Data collected for each run.

std::vector<std::string> progressPropertyNames

Names of each of the planner progress properties reported by planner

std::vector<RunProgressData> runsProgressData

For each run of the planner, this stores the set of planner progress data reported by the planner

RunProperties common

Some common properties for all the runs.

struct Request

Representation of a benchmark request.

Public Functions

inline Request(double maxTime = 5.0, double maxMem = 4096.0, unsigned int runCount = 100, double timeBetweenUpdates = 0.05, bool displayProgress = true, bool saveConsoleOutput = true, bool simplify = true)

Constructor that provides default values for all members.

Public Members

double maxTime

the maximum amount of time a planner is allowed to run (seconds); 5.0 by default

double maxMem

the maximum amount of memory a planner is allowed to use (MB); 4096.0 by default

unsigned int runCount

the number of times to run each planner; 100 by default If set to 0, then run each planner as many times as possible with maxTime total time limit

double timeBetweenUpdates

When collecting time-varying data from a planner during its execution, the planner’s progress will be queried every timeBetweenUpdates seconds.

bool displayProgress

flag indicating whether progress is to be displayed or not; true by default

bool saveConsoleOutput

flag indicating whether console output is saved (in an automatically generated filename); true by default

bool simplify

flag indicating whether simplification should be applied to path; true by default

struct RunProperties : public std::map<std::string, std::string>

The data collected from a run of a planner is stored as key-value pairs.

struct Status

This structure contains information about the activity of a benchmark instance. If the instance is running, it is possible to find out information such as which planner is currently being tested or how much.

Public Members

bool running = {false}

Flag indicating whether benchmarking is running.

unsigned int activeRun = {0}

The number of the run currently being executed.

double progressPercentage = {0.}

Total progress (0 to 100)

std::string activePlanner

The name of the planner currently being tested.