Previous: Test Policy Home: Next: Local Regression Tests

Test Matrix

The Boost Test Matrix is an automated testing system that runs tests on Boost libraries across a wide range of platforms, compiler versions, and configurations. Its primary purpose is to ensure that the libraries work correctly under various conditions and to identify any compatibility issues.

The Test Matrix includes tests run on different operating systems (Windows, Linux, macOS) and with various compilers (such as GCC, Clang, MSVC). This diversity helps in catching issues that might only appear in specific environments.

For information on running regression tests locally, refer to Local Regression Tests.

Regression Dashboard

The results of library tests are published on the Boost Regression Testing Dashboard:

Version Results Issues

Develop branch

Summary

Unresolved Issues

Master branch

Summary

Unresolved Issues

This dashboard is publicly accessible and provides detailed information about the test results for most libraries.

Handling Test Failures

If your contribution causes test failures, it’s expected that you take responsibility for fixing them. This might involve making code adjustments or collaborating with others if the issue is complex, or affects multiple parts of Boost.

On the dashboard, you will find a matrix of test results. The columns represent different test runners (which correspond to different platforms, compiler versions, etc.), and the rows represent individual libraries.

When you click on the Dashboard URL, you will see the test Summary.

test matrix summary

The table is large, and you will need to scroll up and down and left and right to view all relevant results. Note that some entries are underlined (even OK entries) which indicates they are links to further information.

test matrix sample

Dashboard Legend

The dashboard uses different colors or labels to indicate the status of the tests (such as 'pass', 'fail', or 'unresolved'). Understanding these labels is key to interpreting the test results correctly. Note that some tests are expected to fail.

test matrix legend

By clicking on specific cells in the matrix, you can view detailed results for a particular library and test runner. This includes information about which tests passed, which failed, and often detailed logs of the test runs.

test matrix sample 2

Example

For example, click on the Boost.Asio entry in the left-hand column, and you will bring up the test matrix for the Asio/Master library.

test matrix asio

Go deeper by clicking on the entries in the left-hand column, which are now individual tests, to bring up the test code. For example, click on any_completion_handler to view its source.

test matrix asio completion handler

Selecting the column headers will bring up the configuration information that is being specified before each test in the column is run.

test matrix teeks

Developer Report

If you select the Report info entry, top left on the left-hand column, you can see some metrics for the overall test process:

test matrix developer report

Boost Build

The Test Matrix uses B2 as its build system. Understanding how to write and modify Jamfiles (Boost.Build scripts) can be helpful for integrating your tests into the matrix.

Test Process Feedback

If you encounter issues with the testing infrastructure or have suggestions for improvement, engage with the Boost community (refer to Getting Involved). The testing process is continually evolving, and contributions to the testing infrastructure itself are valuable.

See Also