= Open Test Reporting
:schemaVersion: 0.2.0
:openTestReportingVersion: 0.2.5

<<xml-formats, XML>> and <<cli-html-report, HTML>> test reporting formats that are agnostic of testing framework and programming language.

== Introduction

This repository contains the definition of and tooling for two new XML-based test reporting formats that are agnostic of any testing framework and programming language yet concrete enough to be consumable by a wide range of downstream tools such as IDEs, build tools, and report generators.
Moreover, it provides a report generator that produces a <<cli-html-report, standalone HTML file>> based on the XML formats.

The <<event-based-format, event-based XML format>> is suitable for writing test events to a file or streaming them over a local socket or network connection.
Instead of having to collect all data for a test only to write it to the report after the test has completed, testing frameworks can emit events as they occur, for example when a test is started or finished.

The <<hierarchical-format, hierarchical XML format>> is meant to be closer to existing representations of test results that users are familiar with.
For example, when executing tests in IDEs, one is usually presented with a tree of tests and their results.
This same structure can be found in files that use this format.
Thus, instead of requiring humans or processing tools to piece together all relevant events for a test from the event-based format, this hierarchical format collects all relevant information in a single place for each test.
Besides being human-readable, this format is easier to transform into an HTML report by a downstream reporting tool or CI server.
Both XML formats are designed to complement each other such that the event-based format can be mechanically converted into the hierarchical one.

This repository contains a <<reference-implementation, reference implementation>> of an API for <<events-api, writing>> event-based XML files (from JVM languages such as Java), as well as a <<cli, CLI tool>> for <<cli-validate, validating>> them, <<cli-convert, converting>> them to the hierarchical format, and generating <<cli-html-report, HTML reports>> from them.
This way, testing frameworks can focus on writing the event-based format; build tools may use the converter to write the hierarchical format or generate HTML reports; and reporting tools can consume whichever format they prefer.

NOTE: Please refer to the appendix for more information about the <<prior-art, motivation behind this project>> and its <<#design-goals, design goals>>.

[#xml-formats]
== XML format specification

Both formats share a set of core elements that is defined in link:schema/src/main/resources/org/opentest4j/reporting/schema/core-{schemaVersion}.xsd[core-{schemaVersion}.xsd].

[#event-based-format]
=== Event-based format

The event-based format is defined in link:schema/src/main/resources/org/opentest4j/reporting/schema/events-{schemaVersion}.xsd[events-{schemaVersion}.xsd].

The following example shows the result of a test run with a single top-level "container" and a "test" child.

[source,xml,subs="attributes+"]
----
<?xml version="1.0" ?>
<e:events
        xmlns="https://schemas.opentest4j.org/reporting/core/{schemaVersion}"
        xmlns:e="https://schemas.opentest4j.org/reporting/events/{schemaVersion}">
    <infrastructure> <!--1-->
        <hostName>wonderland</hostName>
        <userName>alice</userName>
    </infrastructure>
    <e:started id="1" name="container" time="2022-02-05T16:30:39.129888Z"/> <!--2-->
    <e:started id="2" name="test" parentId="1" time="2022-02-05T16:30:39.137022Z"/> <!--3-->
    <e:finished id="2" time="2022-02-05T16:30:39.143013Z"> <!--4-->
        <result status="SUCCESSFUL"/>
    </e:finished>
    <e:finished id="1" time="2022-02-05T16:30:39.143292Z"> <!--5-->
        <result status="SUCCESSFUL"/>
    </e:finished>
</e:events>
----
<1> attributes of the infrastructure the tests were run on
<2> start event of "container" with timestamp
<3> start event of "test" with timestamp and reference to its parent "container"
<4> finished event of "test" with timestamp and result status
<5> finished event of "container" with timestamp and result status

This file contains 4 events: 2 started and 2 finished ones.
The test framework reporting these events was able to write them as they occurred.
In particular, it did not have to wait for all children of "container" to be finished.

[#hierarchical-format]
=== Hierarchical format

The event-based format is defined in link:schema/src/main/resources/org/opentest4j/reporting/schema/hierarchy-{schemaVersion}.xsd[hierarchy-{schemaVersion}.xsd].

The following example shows the result of converting the above event-based report into the hierarchical format.

[source,xml,subs="attributes+"]
----
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<h:execution xmlns:h="https://schemas.opentest4j.org/reporting/hierarchy/{schemaVersion}"
             xmlns="https://schemas.opentest4j.org/reporting/core/{schemaVersion}">
    <infrastructure> <!--1-->
        <hostName>wonderland</hostName>
        <userName>alice</userName>
    </infrastructure>
    <h:root duration="PT0.013404S" name="container" start="2022-02-05T16:30:39.129888Z"> <!--2-->
        <result status="SUCCESSFUL"/>
        <h:child duration="PT0.005991S" name="test" start="2022-02-05T16:30:39.137022Z"> <!--3-->
            <result status="SUCCESSFUL"/>
        </h:child>
    </h:root>
</h:execution>
----
<1> attributes of the infrastructure the tests were run on (same as in the event-based format)
<2> root node of "container" with timestamp, duration, and result status
<3> child node of "test" with timestamp, duration, and result status

[#xml-extensions]
=== Schema extensions

All schema definitions mentioned so far are language-agnostic.

==== Java

In order to report Java-specific attributes, e.g. the class or method name of a test, an extension schema is defined in link:schema/src/main/resources/org/opentest4j/reporting/schema/java-{schemaVersion}.xsd[java-{schemaVersion}.xsd].

[TIP]
====
Test frameworks are encouraged to define their own similar framework-specific extensions if they want to report additional information that is not suitable to be added to the core namespace.
See the link:https://github.com/junit-team/junit5/blob/61578038ba1e7f4ea1059aea3e0589581991f531/junit-platform-reporting/src/main/resources/org/junit/platform/reporting/open/xml/junit.xsd[JUnit project] for an example of how to define an extension schema.
====

==== Git

In order to include Git-specific metadata, e.g. the commit hash or branch name, an extension schema is defined in link:schema/src/main/resources/org/opentest4j/reporting/schema/git-{schemaVersion}.xsd[git-{schemaVersion}.xsd].

[WARNING]
====
If you're implementing support for including Git information in an XML report, you must ensure to strip any credentials from the `originUrl` before writing it to the XML file.
Moreover, since Git status output may include file names that contain personally identifiable or otherwise sensitive information, it is recommended to include Git-specific metadata in the XML report only if the requesting user has opted in explicitly.
====

[#reference-implementation]
== Reference implementation

While the reporting formats are language-agnostic, the reference implementation is written in Java due to being the language its authors (the JUnit team) are most familiar with.

This repository contains the following subprojects:

schema::
    XML schema definitions of both formats
events::
    <<events-api, Java API for writing the event-based format>> without having to deal with Java's XML APIs
cli::
    <<cli, Command-line interface>> (CLI) for validating both formats, converting from the event-based to the hierarchical format, and <<cli-html-report, generating HTML reports>>
tooling-core::
    Java API for validating both formats, converting from the event-based to the hierarchical format (suitable for inclusion in build tools and reporting tools), and generating HTML reports
tooling-spi::
    Java Service Provider Interface (SPI) for the <<html-report-extension, extending the HTML report>>
html-report::
    Vue.js-based HTML report single-page application (SPA)
sample-project::
    Sample project that demonstrates how to enable JUnit's event-based output and set up Gradle to convert it to the hierarchical format and generate an HTML report

[#events]

[#events-api]
=== API for writing event-based format

Testing frameworks that run on the JVM can use the API provided by the _events_ subprojects as follows.

[source,java]
----
import org.opentest4j.reporting.events.api.DocumentWriter;
import org.opentest4j.reporting.events.api.NamespaceRegistry;
import org.opentest4j.reporting.events.core.CoreFactory;
import org.opentest4j.reporting.events.root.Events;
import org.opentest4j.reporting.schema.Namespace;

import java.nio.file.Paths;
import java.time.Instant;

import static org.opentest4j.reporting.events.core.CoreFactory.*;
import static org.opentest4j.reporting.events.core.Result.Status.SUCCESSFUL;
import static org.opentest4j.reporting.events.root.RootFactory.finished;
import static org.opentest4j.reporting.events.root.RootFactory.started;

public class DocumentWriterSample {

    public static void main(String[] args) throws Exception {

        NamespaceRegistry namespaceRegistry = NamespaceRegistry.builder(Namespace.REPORTING_CORE) // <1>
                .add("e", Namespace.REPORTING_EVENTS) //
                .add("java", Namespace.REPORTING_JAVA) //
                .build();

        try (DocumentWriter<Events> writer = Events.createDocumentWriter(namespaceRegistry, Paths.get("events.xml"))) {
            writer.append(infrastructure(), infrastructure -> infrastructure // <2>
                    .append(userName("alice")) //
                    .append(hostName("wonderland")));
            writer.append(started("1", Instant.now(), "container")); // <3>
            writer.append(started("2", Instant.now(), "test"), started -> started.withParentId("1")); // <4>
            writer.append(finished("2", Instant.now()), finished -> finished.append(CoreFactory.result(SUCCESSFUL))); // <5>
            writer.append(finished("1", Instant.now()), finished -> finished.append(CoreFactory.result(SUCCESSFUL))); // <6>
        }
    }
}
----
<1> create a registry of all namespaces used in the document along with their prefixes
<2> report infrastructure attributes
<3> start event of "container" with timestamp
<4> start event of "test" with timestamp and reference to its parent "container"
<5> finished event of "test" with timestamp and result status
<6> finished event of "container" with timestamp and result status

==== Streaming to a Writer

For use cases where you need to stream events over a socket or network connection instead of writing to a file, you can use a `Writer` instance:

[source,java]
----
import org.opentest4j.reporting.events.api.DocumentWriter;
import org.opentest4j.reporting.events.api.NamespaceRegistry;
import org.opentest4j.reporting.events.core.CoreFactory;
import org.opentest4j.reporting.events.root.Events;
import org.opentest4j.reporting.schema.Namespace;

import java.io.OutputStreamWriter;
import java.time.Instant;

import static org.opentest4j.reporting.events.core.CoreFactory.*;
import static org.opentest4j.reporting.events.core.Result.Status.SUCCESSFUL;
import static org.opentest4j.reporting.events.root.RootFactory.finished;
import static org.opentest4j.reporting.events.root.RootFactory.started;

public class DocumentWriterStreamingSample {

	public static void main(String[] args) throws Exception {

		NamespaceRegistry namespaceRegistry = NamespaceRegistry.builder(Namespace.REPORTING_CORE) // <1>
				.add("e", Namespace.REPORTING_EVENTS) //
				.add("git", Namespace.REPORTING_GIT) //
				.add("java", Namespace.REPORTING_JAVA) //
				.build();

		// Example: streaming events to System.out (could be a socket, network stream, etc.)
		try (DocumentWriter<Events> writer = Events.createDocumentWriter(namespaceRegistry,
			new OutputStreamWriter(System.out))) {
			writer.append(infrastructure(), infrastructure -> infrastructure // <2>
					.append(hostName("wonderland")) //
					.append(userName("alice")));
			writer.append(started("1", Instant.now(), "container")); // <3>
			writer.append(started("2", Instant.now(), "test"), started -> started.withParentId("1")); // <4>
			writer.append(finished("2", Instant.now()), finished -> finished.append(CoreFactory.result(SUCCESSFUL))); // <5>
			writer.append(finished("1", Instant.now()), finished -> finished.append(CoreFactory.result(SUCCESSFUL))); // <6>
		}
	}
}
----

This allows you to write test events as they occur, without having to collect all data before writing.

TIP: See the link:https://github.com/junit-team/junit5/blob/86539a286b23b9dd192f3e54bc10e3c95f87773f/junit-platform-reporting/src/main/java/org/junit/platform/reporting/open/xml/OpenTestReportGeneratingListener.java[JUnit project] for an example of how to write an event-based report.

[#cli]
=== CLI tool

The CLI tool provided by the _cli_ subprojects provides subcommands for <<cli-validate, validating>> both formats, <<cli-convert, converting>> from the event-based to the hierarchical format, and generating an <<cli-html-report, HTML report>>.

==== Installation

You can download the CLI tool from Maven Central using the https://repo1.maven.org/maven2/org/opentest4j/reporting/open-test-reporting-cli/{openTestReportingVersion}/open-test-reporting-cli-{openTestReportingVersion}-standalone.jar[`org.opentest4j.reporting:open-test-reporting-cli:{openTestReportingVersion}:standalone`] coordinates.

Alternatively, you can use https://www.jbang.dev/[JBang]:

[source,console,subs="attributes+"]
----
$ jbang org.opentest4j.reporting:open-test-reporting-cli:{openTestReportingVersion}:standalone --help
----

[#cli-validate]
==== Validate XML files

[source,console,subs="attributes+"]
----
$ java -jar open-test-reporting-cli-{openTestReportingVersion}.jar validate events.xml
ℹ️ Valid: events.xml
----

NOTE: Please run `validate --help` for more information.

[#cli-convert]
==== Convert event-based to hierarchical format

[source,console],subs="attributes+"]
----
$ java -jar open-test-reporting-cli-{openTestReportingVersion}.jar convert events.xml
ℹ️ Converted events.xml to file:///some/path/hierarchy.xml
----

NOTE: Please run `convert --help` for more information.

[#cli-html-report]
==== HTML report

The HTML report is a single-page application (SPA) that displays the test results in a tree-like structure and allows users to view details about each test/container.
It can be generated as follows using the <<cli, CLI>>:

[source,console,subs="attributes+"]
----
$ java -jar open-test-reporting-cli-{openTestReportingVersion}.jar html-report --output open-test-report.html events-1.xml hierarchical.xml
ℹ️ Wrote HTML report to file:///some/path/open-test-report.html
----

NOTE: Please run `html-report --help` for more information.

The command takes one or multiple XML files in the event-based or hierarchical format as input and generates a single HTML file that can be opened in any modern web browser without requiring a server.

.Sample HTML report where all tests pass (light theme)
--
image::documentation/html-report-successful.png[]
--

.Sample HTML report with a failing test (dark theme)
--
image::documentation/html-report-failed.png[]
--


[#html-report-extension]
===== Extending the HTML report

Since the XML formats are <<xml-extensions, extensible>>, the HTML report can be extended to display additional information that is not part of the core schemas.
In order to do so, the _tooling-spi_ subproject defines a Java Service Provider Interface (SPI) that allows developers to implement a custom extension that is automatically picked up by the HTML report generator when it is on the classpath/module path.
Please refer to the Javadoc of the `org.opentest4j.reporting.tooling.spi.htmlreport.Contributor` interface for more information.

TIP: See the link:https://github.com/junit-team/junit5/blob/86539a286b23b9dd192f3e54bc10e3c95f87773f/junit-platform-reporting/src/main/java/org/junit/platform/reporting/open/xml/JUnitContributor.java[JUnit project] for an example of how to implement a `Contributor`.


[appendix, #prior-art]
== Prior art

In the Java ecosystem, Ant originally defined an XML-based reporting format for tests.
Other build tools like Maven and Gradle have adopted the format and, in the case of Maven Surefire, later made changes to add additional data.
Many build servers know how to parse the XML-based format, and even non-Java tools sometimes support it.
However, it’s based on the concept of test classes and methods, so using it for frameworks and tools where those elements are not present is awkward at best.
Moreover, it does not support nested structures beyond a simple parent-child relationship.
Finally, it is not extensible: no additional attributes can be added without the risk of breaking existing tools.

For those reasons, many testing frameworks such as TestNG and Spock have defined their own reporting formats.
This has given them the flexibility they need, but the number of tools that can parse, display, or transform their custom formats is very limited.

To overcome the limitations described above, this document defines a new format for test reporting.
Its goal is to be platform-agnostic so that as many testing frameworks as possible can benefit from it.
Moreover, it is designed to be extensible so new data can be added as needed, without breaking consumers.
However, all well-known attributes are properly defined so it’s consumable by downstream reporting tools.

[appendix, #design-goals]
== Design goals

human-readable::
The new format needs to be human-readable so it can be inspected without requiring another tool.
A format that represents the test tree via nesting is easier to understand than a flat list of events that reference their parents via IDs.

streamable::
Writing report files during test execution should not require keeping state about currently running tests, etc. in memory.
Instead, the new format should allow tools to write events to a file as they occur.

machine-readable::
At the same time, the format must be machine-readable so it can be easily parsed and transformed on a variety of platforms.

schema-aware::
Implementors and tools should be able to validate a given document against a well-known schema.

extensible::
Adding additional language- or framework-specific attributes to nodes must be possible without breaking backwards compatibility.

=== XML vs. JSON

JSON is less verbose than XML, but the latter provides more expressive ways to define schemas.
Moreover, XML has typed extensions built-in via the use of multiple schemas.
Thus, the new formats use XML with accompanying XML schemas.
