02
Oct

Cobertura Reports for Integration Tests and Unit Tests in Apache Sling

Problem

The past weekend I’ve been struggling to finalize a Maven build that would execute unit tests and integration tests and then aggregate the code coverage reports from these 2 tests.

The task seemed pretty easy at the beginning, especially after checking out the new integration testing tools built in Sling. The sample project is quite straight forward, easy to understand and easy to get started with integration tests.

However, there were a few differences in the way I wanted to run the tests. In Apache Sling integration tests are written in separate projects. In my case, I wanted to create the infrastructure to write and execute the integration tests as part of each OSGI bundle; in this way I would be able to generate a more complete code coverage report, aggregating the results from unit tests with the results from the integration tests.

Solution

  1. Create a special maven profile to execute the unit test : “integration-tests”
  2. Get the code from the integration-tests Sling project sample and integrate it into the profile.
  3. Configure the additional sling bundles that need to be deployed during the integration tests.
  4. Configure the bundle to be tested so that it gets deployed on Sling, along with the other bundles.
  5. Separate executing of unit-tests from integration-tests
  6. Add Cobertura build step in order to compute the code coverage
  7. Aggregate the unit-test reports with the integration-tests reports into a unified code coverage report.

Step by step details

Step 1. Add the new maven profile

       <profile>
            <id>integration-tests</id>
        </profile>

Step 2. Get the Apache Sling testing projects

 svn co http://svn.apache.org/repos/asf/sling/trunk/testing/

Step 3. Copy the build directives and the dependencies from the integration-tests project, into the “integration-tests” profile you’ve created at step 1.
Then update the configuration for maven-failsafe-plugin to include the bundles you need for the integration tests. I.e.

<sling.additional.bundle.1>org.apache.sling.junit.core</sling.additional.bundle.1>
<sling.additional.bundle.2>org.apache.sling.testing.tools</sling.additional.bundle.2>
<sling.additional.bundle.3>org.apache.sling.junit.remote</sling.additional.bundle.3>
... others

Step 4. Add the bundle to be tested into the list of additional bundles b/c the integration tests are for it. Simply update the configuration for build-helper-maven-plugin.

    <plugin>
        <!-- Find free ports to run our server -->
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <version>1.7</version>
        <executions>
            <execution>
                <id>reserve-server-port</id>
                <goals>
                    <goal>reserve-network-port</goal>
                </goals>
                <phase>process-resources</phase>
                <configuration>
                    <portNames>
                        <portName>http.port</portName>
                    </portNames>
                </configuration>
            </execution>
 
            <!-- include integration tests folder when compiling tests -->
            <execution>
                <phase>generate-test-sources</phase>
                <goals>
                    <goal>add-test-source</goal>
                </goals>
                <configuration>
                    <sources>
                        <source>src/integration-test/java</source>
                    </sources>
                </configuration>
            </execution>
 
        </executions>
    </plugin>

Step 5. Separate executing of unit-tests from integration-tests. There are several ways to achieve this, what I’m describing here is just what I’ve chosen.
Update maven folder structure:

-src
  - integration-test
  - main
  - test

Use JUnit4 Test @Category(IntegrationTest.class) annotation to differentiate the integration tests from the unit tests. Even though they dwell in different directories, it made it easier for me to configure Maven build in this way.

Update the configuration for maven-surefire-plugin:

    <!-- ignore integration tests groups -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.12.4</version>
        <dependencies>
            <dependency>
                <groupId>org.apache.maven.surefire</groupId>
                <artifactId>surefire-junit47</artifactId>
                <version>2.12.4</version>
            </dependency>
        </dependencies>
        <configuration>
            <excludedGroups>org.mediacenter.testing.IntegrationTest</excludedGroups>
        </configuration>
    </plugin>

Update the configuration for maven-failsafe-plugin:

    <!-- execute only groups marked with IntegrationTest -->
    <groups>org.mediacenter.testing.IntegrationTest</groups>

Step 6. Add Cobertura build step in order to compute the code coverage. This step was a bit tricky, b/c I had to compile the Cobertura JAR for OSGI; but thanks to the good examples from Apache Felix, it was easy to get it done. The Cobertura dependency needs to be added to the list of additional bundles to be installed during integration tests.

    <!-- Cobertura OSGI bundle used to compute code coverage during integration tests -->
    <dependency>
        <groupId>net.sourceforge.cobertura</groupId>
        <artifactId>cobertura</artifactId>
        <version>1.9.4.1</version>
        <classifier>osgi</classifier>
    </dependency>
    ...
   <sling.additional.bundle.8>cobertura</sling.additional.bundle.8>

Then add the cobertura-it-maven-plugin

    <!-- cobertura hook to calculate code coverage -->
    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>cobertura-it-maven-plugin</artifactId>
        <version>2.5</version>
        <configuration>
            <formats>
                <format>xml</format>
            </formats>
            <check>
                <haltOnFailure>false</haltOnFailure>
            </check>
        </configuration>
        <executions>
            <execution>
                <id>cobertura-clean</id>
                <phase>clean</phase>
                <goals>
                    <goal>clean</goal>
                </goals>
            </execution>
            <execution>
                <id>cobertura-instrument</id>
                <phase>process-classes</phase>
                <goals>
                    <goal>instrument</goal>
                </goals>
            </execution>
            <execution>
                <id>cobertura-check-only</id>
                <phase>verify</phase>
                <goals>
                    <goal>check-only</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

Step 7. Aggregate the unit-test reports with the integration-tests reports into a unified code coverage report.
For unit tests, there is a small trick to unified the reports. Make sure to dump reports into the same place and they get merged. To do this, update maven-failsafe-plugin configuration:

    <!-- show IT results together with the unit tests results -->
    <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>

For integration tests, I had to use an ant task from Cobertura to generate the reports:

    <!-- generate the code coverage report after integration tests through the ant task -->
    <plugin>
        <artifactId>maven-antrun-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>net.sourceforge.cobertura</groupId>
                <artifactId>cobertura</artifactId>
                <version>1.9.4.1</version>
            </dependency>
            <dependency>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
                <version>1.2.9</version>
            </dependency>
        </dependencies>
        <executions>
            <execution>
                <phase>post-integration-test</phase>
                <id>cobertura-report</id>
                <configuration>
                    <tasks>
                        <taskdef classpathref="maven.runtime.classpath"
                                 resource="tasks.properties"/>
                        <mkdir dir="${project.build.directory}/site/cobertura"/>
                        <cobertura-report format="xml"
                                          datafile="${project.build.directory}/cobertura/cobertura.ser"
                                          destdir="${project.build.directory}/site/cobertura">
                        </cobertura-report>
                        <cobertura-report format="html"
                                          datafile="${project.build.directory}/cobertura/cobertura.ser"
                                          destdir="${project.build.directory}/site/cobertura">
                        </cobertura-report>
                    </tasks>
                </configuration>
                <goals>
                    <goal>run</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

That’s it. I tried to describe briefly about 14 hours of work.

Now, after a successful build in Jenkins ( mvn clean verify site -Pintegration-tests ) I was able to get the aggregated reports for the unit test and integration tests.