README
ament_cmake_google_benchmark
This package contains logic for invoking Google Benchmark tests and is comprised of three main parts:
The
ament_add_google_benchmark*
CMake macros.
These macros are used in ament packages to compile Google Benchmark executables, run Google Benchmark tests with CTest, or both.The
run_and_convert.py
test wrapper.
This script is used to convert the Google Benchmark JSON results to a custom format which can be consumed by the Jenkins Benchmark plugin. The script also combines the generated results with elements from an overlay file, which may contain additional context for the benchmarks, such as descriptions or value thresholds.The
benchmark_schema.json
output schema.
This schema describes the output format for therun_and_convert.py
test wrapper, and should be used by Jenkins in parsing the results for aggregation and threshold checks. It is also the schema used by the overlay file.
Examples
The source file format is well-described in the Google Benchmark README. Here is a simple example:
#include <benchmark/benchmark.h>
static void increment_perf(benchmark::State & state)
{
size_t i = 0;
for (auto _ : state) {
i++;
}
}
BENCHMARK(increment_perf);
To compile and run this benchmark, the following is added to the CMakeLists.txt
:
if(BUILD_TESTING)
find_package(ament_cmake_google_benchmark REQUIRED)
ament_add_google_benchmark(simple_benchmark test/simple_benchmark.cpp)
endif()
Jenkins can also trigger a build warning when the metrics exceed predefined thresholds. Here is an example overlay file to sets a threshold of 0.6% deviation above or below the average metric from all previous Jenkins job builds which include that metric:
{
"package_name.simple_benchmark": {
"increment_perf": {
"real_time": {
"thresholds": [
{
"method": "percentageaverage",
"percentage": 0.6
}
]
}
}
}
}
This overlay file can be generated at build time or checked in as a static file. The location of the file is
communicated to this package using the AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY
CMake variable. For example:
if(BUILD_TESTING)
set(AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY test/benchmark_thresholds.json)
...
endif()