Summary
We encountered the issue that step definitions which throw an exception are reported as passed, when using the Boost.Test framework and cucumber-cpp.
Expected Behavior
Our expectation was that a step which throws an exception (for whatever reason) should be reported as a failed step by cucumber-cpp.
Current Behavior
The step is reported as passed. The rest of the step definition after the exception gets thrown, doesn't get executed.
Possible Solution
Not familiar with Boost.Test nor cucumber-cpp source code, but while debugging this, I noticed the following:
The exception is caught by unit_test_monitor_t::execute_and_translate(...) in unit_test_monitor.ipp and gets reported to test observers.
However, the BoostDriver only considers the test log for the decision to rate a step as success or failure, but exceptions don't show up in this log for some reason.
We could successfully trigger a failed step by adding something along the following lines after runWithMasterSuite() (needs #include <boost/test/results_collector.hpp>):
auto results = ::boost::unit_test::results_collector.results(testCase->p_id);
if (!results.passed())
{
std::string description = "Test aborted/failed with result code " + std::to_string(results.result_code());
return InvokeResult::failure(description.c_str());
}
Summary
We encountered the issue that step definitions which throw an exception are reported as passed, when using the Boost.Test framework and cucumber-cpp.
Expected Behavior
Our expectation was that a step which throws an exception (for whatever reason) should be reported as a failed step by cucumber-cpp.
Current Behavior
The step is reported as passed. The rest of the step definition after the exception gets thrown, doesn't get executed.
Possible Solution
Not familiar with Boost.Test nor cucumber-cpp source code, but while debugging this, I noticed the following:
The exception is caught by
unit_test_monitor_t::execute_and_translate(...)in unit_test_monitor.ipp and gets reported to test observers.However, the BoostDriver only considers the test log for the decision to rate a step as success or failure, but exceptions don't show up in this log for some reason.
We could successfully trigger a failed step by adding something along the following lines after
runWithMasterSuite()(needs#include <boost/test/results_collector.hpp>):