8000 Merge branch 'develop' of https://github.com/SeanCline/Catch into dev… · log4cplus/Catch@1cbc4f2 · GitHub
[go: up one dir, main page]

Skip to content

Commit 1cbc4f2

Browse files
committed
Merge branch 'develop' of https://github.com/SeanCline/Catch into develop
2 parents 92f0836 + 6e99695 commit 1cbc4f2

File tree

3 files changed

+151
-66
lines changed

3 files changed

+151
-66
lines changed

docs/build-systems.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,50 @@
11
# Integration with build systems
22

3+
Build Systems may refer to low-level tools, like CMake, or larger systems that run on servers, like Jenkins or TeamCity. This page will talk about both.
4+
5+
# Continuous Integration systems
6+
7+
Probably the most important aspect to using Catch with a build server is the use of different reporters. Catch comes bundled with three reporters that should cover the majority of build servers out there - although adding more for better integration with some is always a possibility (as has been done with TeamCity).
8+
9+
Two of these reporters are built in (XML and JUnit) and the third (TeamCity) is included as a separate header. It's possible that the other two may be split out in the future too - as that would make the core of Catch smaller for those that don't need them.
10+
11+
## XML Reporter
12+
```-r xml```
13+
14+
The XML Reporter writes in an XML format that is specific to Catch.
15+
16+
The advantage of this format is that it corresponds well to the way Catch works (especially the more unusual features, such as nested sections) and is a fully streaming format - that is it writes output as it goes, without having to store up all its results before it can start writing.
17+
18+
The disadvantage is that, being specific to Catch, no existing build servers understand the format natively. It can be used as input to an XSLT transformation that could covert it to, say, HTML - although this loses the streaming advantage, of course.
19+
20+
## JUnit Reporter
21+
```-r junit```
22+
23+
The JUnit Reporter writes in an XML format that mimics the JUnit ANT schema.
24+
25+
The advantage of this format is that the JUnit Ant schema is widely understood by most build servers and so can usually be consumed with no additional work.
26+
27+
The disadvantage is that this schema was designed to correspond to how JUnit works - and there is a significant mismatch with how Catch works. Additionally the format is not streamable (because opening elements hold counts of failed and passing tests as attributes) - so the whole test run must complete before it can be written.
28+
29+
## TeamCity Reporter
30+
```-r teamcity```
31+
32+
The TeamCity Reporter writes TeamCity service messages to stdout. In order to be able to use this reporter an additional header must also be included.
33+
34+
```catch_reporter_teamcity.hpp``` can be found in the ```include\reporters``` directory. It should be included in the same file that ```#define```s ```CATCH_CONFIG_MAIN``` or ```CATCH_CONFIG_RUNNER```. The ```#include``` should be placed after ```#include```ing Catch itself.
35+
36+
e.g.:
37+
38+
```
39+
#define CATCH_CONFIG_MAIN
40+
#include "catch.hpp"
41+
#include "catch_reporter_teamcity.hpp"
42+
```
43+
44+
Being specific to TeamCity this is the best reporter to use with it - but it is completely unsuitable for any other purpose. It is a streaming format (it writes as it goes) - although test results don't appear in the TeamCity interface until the completion of a suite (usually the whole test run).
45+
46+
# Low-level tools
47+
348
## CMake
449

550
You can use the following CMake script to automatically fetch Catch from github and configure it as an external project:

include/internal/catch_impl.hpp

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,8 +88,6 @@ namespace Catch {
8888
Matchers::Impl::StdString::EndsWith::~EndsWith() {}
8989

9090
void Config::dummy() {}
91-
92-
INTERNAL_CATCH_REGISTER_LEGACY_REPORTER( "xml", XmlReporter )
9391
}
9492

9593
#ifdef __clang__

include/reporters/catch_reporter_xml.hpp

Lines changed: 106 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -13,107 +13,117 @@
1313
#include "../internal/catch_capture.hpp"
1414
#include "../internal/catch_reporter_registrars.hpp"
1515
#include "../internal/catch_xmlwriter.hpp"
16+
#include "../internal/catch_timer.h"
1617

1718
namespace Catch {
18-
class XmlReporter : public SharedImpl<IReporter> {
19+
class XmlReporter : public StreamingReporterBase {
1920
public:
20-
XmlReporter( ReporterConfig const& config ) : m_config( config ), m_sectionDepth( 0 ) {}
21+
XmlReporter( ReporterConfig const& _config )
22+
: StreamingReporterBase( _config ),
23+
m_sectionDepth( 0 )
24+
{}
2125

26+
virtual ~XmlReporter();
27+
2228
static std::string getDescription() {
2329
return "Reports test results as an XML document";
2430
}
25-
virtual ~XmlReporter();
26-
27-
private: // IReporter
2831

29-
virtual bool shouldRedirectStdout() const {
30-
return true;
32+
public: // StreamingReporterBase
33+
virtual ReporterPreferences getPreferences() const {
34+
ReporterPreferences prefs;
35+
prefs.shouldRedirectStdOut = true;
36+
return prefs;
3137
}
3238

33-
virtual void StartTesting() {
34-
m_xml.setStream( m_config.stream() );
35-
m_xml.startElement( "Catch" );
36-
if( !m_config.fullConfig()->name().empty() )
37-
m_xml.writeAttribute( "name", m_config.fullConfig()->name() );
39+
virtual void noMatchingTestCases( std::string const& s ) {
40+
StreamingReporterBase::noMatchingTestCases( s );
3841
}
3942

40-
virtual void EndTesting( const Totals& totals ) {
41-
m_xml.scopedElement( "OverallResults" )
42-
.writeAttribute( "successes", totals.assertions.passed )
43-
.writeAttribute( "failures", totals.assertions.failed )
44-
.writeAttribute( "expectedFailures", totals.assertions.failedButOk );
45-
m_xml.endElement();
43+
virtual void testRunStarting( TestRunInfo const& testInfo ) {
44+
StreamingReporterBase::testRunStarting( testInfo );
45+
m_xml.setStream( stream );
46+
m_xml.startElement( "Catch" );
47+
if( !m_config->name().empty() )
48+
m_xml.writeAttribute( "name", m_config->name() );
4649
}
4750

48-
virtual void StartGroup( const std::string& groupName ) {
51+
virtual void testGroupStarting( GroupInfo const& groupInfo ) {
52+
StreamingReporterBase::testGroupStarting( groupInfo );
4953
m_xml.startElement( "Group" )
50-
.writeAttribute( "name", groupName );
54+
.writeAttribute( "name", groupInfo.name );
5155
}
5256

53-
virtual void EndGroup( const std::string&, const Totals& totals ) {
54-
m_xml.scopedElement( "OverallResults" )
55-
.writeAttribute( "successes", totals.assertions.passed )
56-
.writeAttribute( "failures", totals.assertions.failed )
57-
.writeAttribute( "expectedFailures", totals.assertions.failedButOk );
58-
m_xml.endElement();
57+
virtual void testCaseStarting( TestCaseInfo const& testInfo ) {
58+
StreamingReporterBase::testCaseStarting(testInfo);
59+
m_xml.startElement( "TestCase" ).writeAttribute( "name", trim( testInfo.name ) );
60+
61+
if ( m_config->showDurations() == ShowDurations::Always )
62+
m_testCaseTimer.start();
5963
}
6064

61-
virtual void StartSection( const std::string& sectionName, const std::string& description ) {
65+
virtual void sectionStarting( SectionInfo const& sectionInfo ) {
66+
StreamingReporterBase::sectionStarting( sectionInfo );
6267
if( m_sectionDepth++ > 0 ) {
6368
m_xml.startElement( "Section" )
64-
.writeAttribute( "name", trim( sectionName ) )
65-
.writeAttribute( "description", description );
69+
.writeAttribute( "name", trim( sectionInfo.name ) )
70+
.writeAttribute( "description", sectionInfo.description );
6671
}
6772
}
68-
virtual void NoAssertionsInSection( const std::string& ) {}
69-
virtual void NoAssertionsInTestCase( const std::string& ) {}
7073

71-
virtual void EndSection( const std::string& /*sectionName*/, const Counts& assertions ) {
72-
if( --m_sectionDepth > 0 ) {
73-
m_xml.scopedElement( "OverallResults" )
74-
.writeAttribute( "successes", assertions.passed )
75-
.writeAttribute( "failures", assertions.failed )
76-
.writeAttribute( "expectedFailures", assertions.failedButOk );
77-
m_xml.endElement();
74+
virtual void assertionStarting( AssertionInfo const& ) { }
75+
76+
virtual bool assertionEnded( AssertionStats const& assertionStats ) {
77+
const AssertionResult& assertionResult = assertionStats.assertionResult;
78+
79+
// Print any info messages in <Info> tags.
80+
if( assertionStats.assertionResult.getResultType() != ResultWas::Ok ) {
81+
for( std::vector<MessageInfo>::const_iterator it = assertionStats.infoMessages.begin(), itEnd = assertionStats.infoMessages.end();
82+
it != itEnd;
83+
++it ) {
84+
if( it->type == ResultWas::Info ) {
85+
m_xml.scopedElement( "Info" )
86+
.writeText( it->message );
87+
} else if ( it->type == ResultWas::Warning ) {
88+
m_xml.scopedElement( "Warning" )
89+
.writeText( it->message );
90+
}
91+
}
7892
}
79-
}
80-
81-
virtual void StartTestCase( const Catch::TestCaseInfo& testInfo ) {
82-
m_xml.startElement( "TestCase" ).writeAttribute( "name", trim( testInfo.name ) );
83-
m_currentTestSuccess = true;
84-
}
8593

86-
virtual void Result( const Catch::AssertionResult& assertionResult ) {
87-
if( !m_config.fullConfig()->includeSuccessfulResults() && assertionResult.getResultType() == ResultWas::Ok )
88-
return;
94+
// Drop out if result was successful but we're not printing them.
95+
if( !m_config->includeSuccessfulResults() && isOk(assertionResult.getResultType()) )
96+
return true;
8997

98+
// Print the expression if there is one.
9099
if( assertionResult.hasExpression() ) {
91100
m_xml.startElement( "Expression" )
92101
.writeAttribute( "success", assertionResult.succeeded() )
102+
.writeAttribute( "type", assertionResult.getTestMacroName() )
93103
.writeAttribute( "filename", assertionResult.getSourceInfo().file )
94104
.writeAttribute( "line", assertionResult.getSourceInfo().line );
95105

96106
m_xml.scopedElement( "Original" )
97107
.writeText( assertionResult.getExpression() );
98108
m_xml.scopedElement( "Expanded" )
99109
.writeText( assertionResult.getExpandedExpression() );
100-
m_currentTestSuccess &= assertionResult.succeeded();
101110
}
102111

112+
// And... Print a result applicable to each result type.
103113
switch( assertionResult.getResultType() ) {
114+
default:
115+
break;
104116
case ResultWas::ThrewException:
105117
m_xml.scopedElement( "Exception" )
106118
.writeAttribute( "filename", assertionResult.getSourceInfo().file )
107119
.writeAttribute( "line", assertionResult.getSourceInfo().line )
108120
.writeText( assertionResult.getMessage() );
109-
m_currentTestSuccess = false;
110121
break;
111122
case ResultWas::FatalErrorCondition:
112123
m_xml.scopedElement( "Fatal Error Condition" )
113124
.writeAttribute( "filename", assertionResult.getSourceInfo().file )
114125
.writeAttribute( "line", assertionResult.getSourceInfo().line )
115126
.writeText( assertionResult.getMessage() );
116-
m_currentTestSuccess = false;
117127
break;
118128
case ResultWas::Info:
119129
m_xml.scopedElement( "Info" )
@@ -126,36 +136,68 @@ namespace Catch {
126136
case ResultWas::ExplicitFailure:
127137
m_xml.scopedElement( "Failure" )
128138
.writeText( assertionResult.getMessage() );
129-
m_currentTestSuccess = false;
130-
break;
131-
case ResultWas::Unknown:
132-
case ResultWas::Ok:
133-
case ResultWas::FailureBit:
134-
case ResultWas::ExpressionFailed:
135-
case ResultWas::Exception:
136-
case ResultWas::DidntThrowException:
137139
break;
138140
}
141+
139142
if( assertionResult.hasExpression() )
140143
m_xml.endElement();
144+
145+
return true;
146+
}
147+
148+
virtual void sectionEnded( SectionStats const& sectionStats ) {
149+
StreamingReporterBase::sectionEnded( sectionStats );
150+
if( --m_sectionDepth > 0 ) {
151+
XmlWriter::ScopedElement e = m_xml.scopedElement( "OverallResults" );
152+
e.writeAttribute( "successes", sectionStats.assertions.passed );
153+
e.writeAttribute( "failures", sectionStats.assertions.failed );
154+
e.writeAttribute( "expectedFailures", sectionStats.assertions.failedButOk );
155+
156+
if ( m_config->showDurations() == ShowDurations::Always )
157+
e.writeAttribute( "durationInSeconds", sectionStats.durationInSeconds );
158+
159+
m_xml.endElement();
160+
}
141161
}
142162

143-
virtual void Aborted() {
144-
// !TBD
163+
virtual void testCaseEnded( TestCaseStats const& testCaseStats ) {
164+
StreamingReporterBase::testCaseEnded( testCaseStats );
165+
XmlWriter::ScopedElement e = m_xml.scopedElement( "OverallResult" );
166+
e.writeAttribute( "success", testCaseStats.totals.assertions.allPassed() );
167+
168+
if ( m_config->showDurations() == ShowDurations::Always )
169+
e.writeAttribute( "durationInSeconds", m_testCaseTimer.getElapsedSeconds() );
170+
171+
m_xml.endElement();
145172
}
146173

147-
virtual void EndTestCase( const Catch::TestCaseInfo&, const Totals&, const std::string&, const std::string& ) {
148-
m_xml.scopedElement( "OverallResult" ).writeAttribute( "success", m_currentTestSuccess );
174+
virtual void testGroupEnded( TestGroupStats const& testGroupStats ) {
175+
StreamingReporterBase::testGroupEnded( testGroupStats );
176+
// TODO: Check testGroupStats.aborting and act accordingly.
177+
m_xml.scopedElement( "OverallResults" )
178+
.writeAttribute( "successes", testGroupStats.totals.assertions.passed )
179+
.writeAttribute( "failures", testGroupStats.totals.assertions.failed )
180+
.writeAttribute( "expectedFailures", testGroupStats.totals.assertions.failedButOk );
181+
m_xml.endElement();
182+
}
183+
184+
virtual void testRunEnded( TestRunStats const& testRunStats ) {
185+
StreamingReporterBase::testRunEnded( testRunStats );
186+
m_xml.scopedElement( "OverallResults" )
187+
.writeAttribute( "successes", testRunStats.totals.assertions.passed )
188+
.writeAttribute( "failures", testRunStats.totals.assertions.failed )
189+
.writeAttribute( "expectedFailures", testRunStats.totals.assertions.failedButOk );
149190
m_xml.endElement();
150191
}
151192

152193
private:
153-
ReporterConfig m_config;
154-
bool m_currentTestSuccess;
194+
Timer m_testCaseTimer;
155195
XmlWriter m_xml;
156196
int m_sectionDepth;
157197
};
158198

199+
INTERNAL_CATCH_REGISTER_REPORTER( "xml", XmlReporter )
200+
159201
} // end namespace Catch
160202

161203
#endif // TWOBLUECUBES_CATCH_REPORTER_XML_HPP_INCLUDED

0 commit comments

Comments
 (0)
0