# Derby Compatibility Tests

- Overview
- Test Setup
- Running the Tests from Ant
- Running the Tests from JUnit

## Overview

This web page describes how to setup and run the Derby Compatibility Tests. These tests stress the compatibility of Derby clients and servers running at different rev levels and on different VMs. We currently test the following combinations:

> ### Clients

> Derby 10.1
> Derby 10.2

> ### Servers

> Derby 10.1
> Derby 10.2

> ### VMs

> jdk1.4
> jdk1.5
> jdk1.6

## Test Setup

To run these tests, you will need to download various VMs and Derby releases. You will also need to edit ant.properties in your home directory, adding variables which point at these VMs and releases. For a complete description of this setup, run the following command, invoking testScript.xml (which lives in the same directory as this web page):

>     ant -buildfile testScript.xml -projecthelp

## Running the Tests from Ant

To run the tests, you will need to do the following:

- mkdir - Create a directory to hold the test output.
- ant - Run ant against the compatibility test script.

Something like the following should work:

>     #! /usr/bin/bash
>     #
>     # Run the compatibility tests.
>     #
>     # $1    directory to hold output, e.g., c:/cygwin/home/aliceBToklas/derby/mainline/testRun
>     # $2    location of derby trunk, e.g., c:/cygwin/home/aliceBToklas/derby/mainline/testRun/trunk

>     outDir=$1
>     trunkDir=$2
>     COMPAT_DIR=$trunkDir/java/testing/org/apache/derbyTesting/functionTests/tests/junitTests/compatibility

>     rm -rf $outDir
>     mkdir $outDir

>     ant -buildfile $COMPAT_DIR/testScript.xml -DoutDir=$outDir

When the tests startup, they defensively bring down the Derby network server in case it was running. Generally, it won't be, and you can expect to see a couple vacuous diagnostics, which you may ignore:

>     _stopServer:
>          [exec] Could not connect to Derby Network Server on host localhost, port 1527.
>          [exec] Result: 1

If the tests succeed, ant will happily report:

>     BUILD SUCCESSFUL

Otherwise, ant will report a failure. In this case, the output directory will hold a file called SUMMARY_COMPAT.txt. This file will identify which test run failed. More specifically, SUMMARY_COMPAT.txt will hold the name of an output file which describes the specific test failures.

## Running the Tests from JUnit

You can also run the tests directly using JUnit. In that case, you need a properties file that tells which combinations to test. The file should be named `compatibilitytest.properties` and it must be placed in the directory from which the tests are started (current working directory).

For example, `compatibilitytest.properties` could look like this:

>     ##
>     ## Example compatibilitytest.properties file
>     ##

>     ##
>     ## Specify which JVMs to test by defining properties named
>     ## jvm.versions, containing the number of JVMs to test, and jvm.N
>     ## (where 0 <= N < ${jvm.versions}), containing a symbolic name for
>     ## the particular JVM to test.
>     ##
>     ## This example will make the test run with Java SE 5 and Java SE 6:
>     ##

>     jvm.versions=2
>     jvm.0=j15lib
>     jvm.1=j16lib

>     ##
>     ## Specify where to find the JVMs. Property names should match the
>     ## symbolic names in the jvm.N properties. The properties should point
>     ## to the jre/lib directory of a JDK.
>     ##

>     j15lib=/usr/jdk/instances/jdk1.5.0/jre/lib
>     j16lib=/usr/jdk/instances/jdk1.6.0/jre/lib

>     ##
>     ## Specify which Derby versions to test. derby.versions contains the
>     ## number of versions to test, derby.versionN (where 0 <= N <
>     ## ${derby.versions}) contains the name of a Derby version.
>     ##
>     ## This example tests four versions of Derby: 10.0.2.1, 10.4.1.3,
>     ## 10.5.3.0 and head of trunk:
>     ##

>     derby.versions=4
>     derby.version0=10.0.2.1
>     derby.version1=10.4.1.3
>     derby.version2=10.5.3.0
>     derby.version3=Trunk

>     ##
>     ## Specify where to find the jars for the Derby versions specified
>     ## above. Property names should match names used in the derby.versionN
>     ## properties.
>     ##

>     10.0.2.1=/opt/db-derby-10.0.2.1-bin/lib
>     10.4.1.3=/opt/db-derby-10.4.1.3-bin/lib
>     10.5.3.0=/opt/db-derby-10.5.3.0-bin/lib
>     Trunk=/code/derby/trunk/jars/sane

>     ##
>     ## We also need to specify whether or not the Derby versions tested
>     ## support the -noSecurityManager option when starting the network
>     ## server. 10.2 and earlier versions don't support it, 10.3 and later
>     ## versions do. Use the version name specified in the derby.versionN
>     ## properties and append "_SA" to the name (short for security
>     ## enabled).
>     ##

>     10.0.2.1_SA=false
>     10.4.1.3_SA=true
>     10.5.3.0_SA=true
>     Trunk_SA=true

>     ##
>     ## Specify the name of the -noSecurityManager option. (Yes, it's
>     ## required, but it will always be the same.)
>     ##

>     test.securityOption=noSecurityManager

>     ##
>     ## Tell the tests where to find junit.jar.
>     ##

>     junit_jar=/usr/share/lib/java/junit.jar

>     ##
>     ## Optional properties to limit the number of combinations to test.
>     ##
>     ## Test only a with a single Derby Network Server version:
>     ##  test.singleServer=N
>     ##
>     ## Test only with a single JVM version on the server side:
>     ##  test.singleServerVM=N
>     ##
>     ## Test only with a single Derby Network Client version:
>     ##  test.singleClient=N
>     ##
>     ## Test only with a single JVM version on the client side:
>     ##  test.singleClientVM=N
>     ##
>     ## The N in the property values is a number that matches the N in one
>     ## of the jvm.N or derby.versionN properties.
>     ##
>     ## Example: Only test combinations that use the trunk version of the
>     ## Network Server, running on a Java 6 virtual machine:
>     ##
>     ## test.singleServer=3
>     ## test.singleServerVM=1
>     ##
>     ## Test only combinations that use the latest Derby version either on
>     ## the server side or on the client side (assuming the derby.versionN
>     ## property with the highest N points to the latest version):
>     ##
>     ## test.latestOnly=true
>     ##

To start the compatibility tests, add the Derby jar files and `junit.jar` to your `CLASSPATH`, and execute the following command:

>     java -Djava.security.policy="<NONE>" junit.textui.TestRunner \
>         org.apache.derbyTesting.functionTests.tests.junitTests.compatibility.CompatibilityCombinations

If all the tests pass, JUnit will report:

>     OK (2 tests)

Otherwise, JUnit will report a failure. In that case, the file `ServerClient_wombat_failed` will contain a list of file names with more information about the failed combinations.