Unit and Integration Testing with Maven, Part 2

Welome Back...

... to the second installment of this little series. After having seen the requirements and hassles when using Maven for testing in the last post, we are now going to list the possible solutions.

Remember, Maven does support different phases for unit and integration tests, but there is only one source directory (usually src/test/java), making it a bit difficult to setup and organize test environments. Hence, we have to use some way to separate both types of test.

Option 1: Separate Module for Integration Tests

In case you are using modules for your project anyways and want to have integration tests which are testing these modules in integration, this is the natural solution: just create another module that depends on the other ones and only contains the integration test sources and resources.

However, if you instead want to integration-test each module individually, this approach just doubles the number of modules, which can be difficult to manage. You could put all integration tests into one big module – but that has other disadvantages, of course.

Since we do not have any sources (only test sources) in that integration-test project, the usual build lifecycle would execute a lot of unneeded steps like copying resources, compiling, testing, creating JAR file etc. We could use the POM packaging type to suppress all of this, but then we have to configure the maven-compiler-plugin to force compilation of test sources.

Independantly of packaging type, we have to do some more configurations:

• Execute Surefire plugin in integration-test phase.
• Prepare integration tests (for instance, start the container and deploy the application) in pre-integration-test phase. This is quite easy with Cargo Maven plugin.
• Shutdown integration tests (stop the container) in post-integration-test phase.

Here is the relevant part of such a POM file:

<build>
  <plugins>

    <!-- *** Compiler plugin: we must force test compile because we're using a 
             pom packaging that doesn't have this lifecycle mapping. -->
    <plugin>
      <artifactId>maven-compiler-plugin</artifactId>
      <executions>
        <execution>
          <goals>
            <goal>testCompile</goal>
          </goals>
        </execution>
      </executions>
    </plugin>

    <!-- *** Surefire plugin: run integration tests *** -->
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <executions>
        <execution>
          <phase>integration-test</phase>
          <goals>
            <goal>test</goal>
          </goals>
        </execution>
      </executions>
    </plugin>

    <!-- *** Cargo plugin: start/stop application server and deploy the ear 
             file before/after integration tests *** -->
    <plugin>
      <groupId>org.codehaus.cargo</groupId>
      <artifactId>cargo-maven2-plugin</artifactId>
      <version>1.0</version>
      <configuration>
       ...
      </configuration>

      <executions>
        <!-- before integration tests are run: start server -->
        <execution>
          <id>start-container</id>
          <phase>pre-integration-test</phase>
          <goals>
            <goal>start</goal>
          </goals>
        </execution>
        <!-- after integration tests are run: stop server -->
        <execution>
          <id>stop-container</id>
          <phase>post-integration-test</phase>
          <goals>
            <goal>stop</goal>
          </goals>
        </execution>
      </executions>
    </plugin>

  </plugins>
</build>

Option 2: Different Source Directories

In this scenario, unit and integration test sources are placed in separate source directories, like src/test/java and src/integrationtest/java. I think this would definitely be the best solution, and it actually should be what Maven supports out of the box. Sadly, Maven does not, and as far as I know Maven 3 won't either :-(

Well, we should be able to configure things this way. For compiling and executing integration tests, you would have to configure Compiler plugin to compile the integration test sources in pre-integration-test phase (using the includes parameter), and then configure a second Surefire execution using testClassesDirectory parameter to point it to the integration test folder.

However, this option seems a bit fragile to me. Even if it works (I haven't checked), the integration test folder would probably not show up in Eclipse when using m2eclipse plugin and other plugins may have issues with this additional test source path as well. That's why I do not recommend this option.

Option 3: Different File Name Patterns

In this scenario, the package or file name is used to distinguish between unit and integration test source files. Let's say, for instance, that all integration test classes start with IT* and hence the filename pattern is **/IT*.java. We'd have to configure the Surefire plugin to execute twice: once in test phase for executing the unit tests only, and another time in integration-test phase to execute, well, the integration tests (and only those)

A common way to do so is this:

• In configuration of Surefire plugin, set skip parameter to true to bypass all tests.
• Add an execution element for unit tests, where skip is set to false and exclude parameter is used to exclude the integration tests.
• Add another execution element for integration tests, where skip is set to false and include parameter is used to just include the integration tests and nothing else.
• Prepare and shutdown of integration tests are like before.

Here is the POM section:

<build>
  <plugins>

    <!-- *** Surefire plugin: run unit and integration tests in 
             separate lifecycle phases, using file name pattern *** -->
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
        <skip>true</skip>
      </configuration>
      <executions>
        <execution>
          <id>unit-test</id>
          <phase>test</phase>
          <goals>
            <goal>test</goal>
          </goals>
          <configuration>
            <skip>false</skip>
            <excludes>
              <exclude>**/IT*.java</exclude>
            </excludes>
          </configuration>
        </execution>

        <execution>
          <id>integration-test</id>
          <phase>integration-test</phase>
          <goals>
            <goal>test</goal>
          </goals>
          <configuration>
            <skip>false</skip>
            <includes>
              <include>**/IT*.java</include>
            </includes>
          </configuration>
        </execution>
      </executions>
    </plugin>

    <!-- *** Cargo plugin: start/stop application server and deploy the ear 
             file before/after integration tests *** -->
    ...

  </plugins>
</build>

There is another approach to achieve the same result:

• In configuration of Surefire plugin, use the exclude parameter to exclude the integration tests when executing unit tests in test phase.
• Add an execution element for integration tests, where exclude is set to any dummy value (to override the default configuration) and include parameter is used to just include the integration tests and nothing else.

It's a bit shorter than the former configuration, but in the end it's a matter of taste. Again, here is the POM snippet:

<build>
  <plugins>

    <!-- *** Surefire plugin: run unit and integration tests in 
             separate lifecycle phases, using file name pattern *** -->
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
        <excludes>
          <exclude>**/IT*.java</exclude>
        </excludes>
      </configuration>
      <executions>
        <execution>
          <id>integration-test</id>
          <phase>integration-test</phase>
          <goals>
            <goal>test</goal>
          </goals>
          <configuration>
            <excludes>
              <exclude>none</exclude>
            </excludes>
            <includes>
              <include>**/IT*.java</include>
            </includes>
          </configuration>
        </execution>
      </executions>
    </plugin>

    <!-- *** Cargo plugin: start/stop application server and deploy the ear 
             file before/after integration tests *** -->
    ...

  </plugins>
</build>

Failsafe Plugin

Instead of using the Surefire plugin with custom filename pattern to distinguish between unit and integration test classes, you should rather use the Failsafe plugin for executing integration tests.

This plugin is a fork of the Surefire plugin designed to run integration tests.
It is used during the integration-test and verify phases of the build lifecycle to execute the integration tests of an application. Other than Surefire plugin, the Failsafe plugin will not fail the build when executing tests thus enabling the post-integration-test phase to execute.

Failsafe plugin has its own naming convention. By default, the Surefire plugin executes **/Test*.java, **/*Test.java, and **/*TestCase.java test classes. In contrast, the Failsafe plugin will look for **/IT*.java, **/*IT.java, and **/*ITCase.java. Did you note that this matches what we used before for our integration tests? ;-)

When using Failsafe, the last POM (of option 3) looks like this:

<build>
  <plugins>

    <!-- *** Surefire plugin: run unit and exclude integration tests *** -->
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
        <excludes>
          <exclude>**/IT*.java</exclude>
        </excludes>
      </configuration>
    </plugin>

    <!-- *** Failsafe plugin: run integration tests *** -->
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>failsafe-maven-plugin</artifactId>
      <version>2.4.3-alpha-1</version>
      <executions>
        <execution>
          <goals>
            <goal>integration-test</goal>
            <goal>verify</goal>
          </goals>
        </execution>
      </executions>
    </plugin>

    <!-- *** Cargo plugin: start/stop application server and deploy the ear 
             file before/after integration tests *** -->
    ...

  </plugins>
</build>

Of course, Failsafe would also work with first option (separate integration test module).

Conclusion

Well, after having shown all the ways that came into my mind, here are my personal "best practices":

• Use the Failsafe plugin to execute integration tests.
• If keeping integration tests in a separate module feels alright, do so (see option 1).
• If you want to have unit and integration tests in the same module, choose a file or package name pattern to distinguish between both, and configure Surefire and Failsafe plugins accordingly.

Update (2010/02/18)

Note that there is a new version 2.5 of failsafe plugin available with changed group id: org.apache.maven.plugins:maven-failsafe-plugin:2.5. See the plugin site for details. Thanks to stug23 for pointing that out!