# Release Notes for Apache Derby 10.10.2.0

These notes describe the difference between Apache Derby release 10.10.2.0 and the preceding release 10.10.1.1.

- Overview
- New Features
- Issues of special concern
- Bug Fixes
- Fixes in the Documentation
- Build Environment
- Verifying Releases

## Overview

The most up to date information about Derby releases can be found on the Derby download page.

Apache Derby is a pure Java relational database engine using standard SQL and JDBC as its APIs. More information about Derby can be found on the Apache web site. Derby functionality includes:

- Embedded engine with JDBC drivers
- Network Server
- Network client JDBC drivers
- Command line tools: ij (SQL scripting), dblook (schema dump) and sysinfo (system info)

Java and JDBC versions supported:

- Java SE 5 and higher with JDBC 3.0, 4.0, 4.1, and 4.2.
- Java ME CDC/Foundation Profile 1.1 with JSR-169 JDBC Optional Package for CDC/Foundation Profile.

## New Features

This is a bug fix release. There are no new features:

## Issues of special concern

Compared with the previous release (10.10.1.1), Derby release 10.10.2.0 introduces the following incompatibilities. These merit your special attention.

- Note for DERBY-6438: Additional permission needed to run Derby network server
- Note for DERBY-6383: After the fix, statement level update triggers defined on specific columns will only fire for the update of any of those specified columns. Additionally, after the fix, statement level update triggers defined at the table level will fire correctly for columns added after the CREATE TRIGGER sql is issued.
- Note for DERBY-6270: Frame injection vulnerability in old versions of the API documentation.
- Note for DERBY-6224: New SQLPermission needed for JDBC driver deregistration on Java SE 8.
- Note for DERBY-6209: When running a server started from the command line with a security manager and a customized policy file, new permissions may be have to be added to it.
- Note for DERBY-6187: Datatype Change in JDBC Metadata
- Note for DERBY-5866: The meaning of SYS.SYSTRIGGERS.CREATIONTIMESTAMP has changed.

### Note for DERBY-6438

#### Summary of Change

Additional permission needed to run Derby network server

#### Symptoms Seen by Applications Affected by Change

An additional java.net.SocketPermission, "listen" was added to the default network server policy file, and the policy file template. The documentation was updated to describe this permission.

#### Incompatibilities with Previous Release

No incompatibilities arise from upgrading a Derby release to one with the additional permissions in the policy file.
However, unless you have this permission, when upgrading to a JVM release that has JDK 7u51 incorporated, your network server may not start if

- you use network server on a port not within the ephemeral ports specified by the OS.
- and
  - you rely on the default derby policy file, or
  - you have a customized policy file without this permission

#### Rationale for Change

An additional permission may be needed in order to bring up the Derby network server after upgrading to JVMs incorporating JDK 7u51. In particular, the startup scripts may fail to boot the network server.

This is a result of the "Better applet networking" changes made for JVM issue 8011787 (not public): the default socket permissions assigned to all code including untrusted code have been changed.
Previously, all code was able to bind any socket type to any port number greater than or equal to 1024. It is still possible to bind sockets to the ephemeral port range on each system. The exact range of ephemeral ports varies from one operating system to another, but it is typically in the high range (such as from 49152 to 65535).
The new restriction is that binding sockets outside of the ephemeral range now requires an explicit permission in the system security policy.

After upgrading to a JVM with these changes, while attempting to boot, the network server may fail and raise the following error:

access denied ("java.net.SocketPermission" "localhost:1527" "listen,resolve") java.security.AccessControlException: access denied ("java.net.SocketPermission" "localhost:1527" "listen,resolve")

To workaround this problem, you must bring up the network server with a security policy which includes the now required missing permission. Instead of booting the network server as:

java org.apache.derby.drda.NetworkServerControl start

boot the network server as follows:

java -Djava.security.manager -Djava.security.policy=${yourPolicyFile} org.apache.derby.drda.NetworkServerControl start

where ${yourPolicyFile} is a file containing a customized version of the policy file described in the Derby Admin Guide section titled Basic Network Server security policy. You must customize that generic policy file to fit your application. In addition, you must add the following permission to the permissions block granted to the ${derby.install.url}derbynet.jar codebase:

permission java.net.SocketPermission "localhost:${port}", "listen";

where ${port} should be replaced by the port number where the network server listens for incoming connection requests. By default, that is port 1527.

For more information on Derby security policies, see the Derby Admin Guide sections titled Network Server security and Running the Network Server under the security manager.

If you are using replication, a similar permission must be granted to the security policy for the slave server. Add the following permission to the ${derby.install.url}derby.jar codebase:

permission java.net.SocketPermission "localhost:${slavePort}", "listen";

where ${slavePort} should be replaced by the port number where the slave server listens for incoming connection requests (typically 4851). For more information on the security policy for the slave server, see the Derby Admin Guide section titled Replication and security.

With this fix, the default policy file has received these required permissions, and the scripts in the bin directory work again. However, it may still be necessary to adjust any customized policy files.

#### Application Changes Required

Applications need not make any change to take advantage of the new permission granted in the default policy file. However, the default policy file may, or may not be sufficient for specific applications and it is better to use your own policy file. If you have your own policy file, you may need to adjust it to add the "listen" permission as appropriate.

### Note for DERBY-6383

#### Summary of Change

After the fix, statement level update triggers defined on specific columns will only fire for the update of any of those specified columns. Additionally, after the fix, statement level update triggers defined at the table level will fire correctly for columns added after the CREATE TRIGGER sql is issued.

#### Symptoms Seen by Applications Affected by Change

Statement level update triggers defined for specific columns fired on update of columns which were not included in the CREATE TRIGGER sql. Also, statement level update triggers defined at table level did not fire for newly added columns.

#### Incompatibilities with Previous Release

Derby releases 10.7.1.4, 10.8.1.2, 10.9.1.0, 10.10.1.1 suffer from DERBY-6383 and will see 1)update statement triggers defined at column(s) level firing for columns not specified in the column list of the "CREATE TRIGGER... AFTER UPDATE... OF" clause 2)update statement triggers defined at table level not firing for columns added to the table after CREATE TRIGGER was issued.

#### Rationale for Change

The fix for the regression identified in DERBY-6383 will now make update statement level triggers adhere to SQL standards.

#### Application Changes Required

In order to fix the buggy triggers created in the above mentioned releases, users will need to manually drop and recreate the statement level update triggers in a release with DERBY-6383 fix. All statement level update triggers, whether they are defined on specific columns or at the table level, should be dropped and recreated.

### Note for DERBY-6270

#### Summary of Change

Frame injection vulnerability in old versions of the API documentation.

#### Symptoms Seen by Applications Affected by Change

Many of the previous releases of Apache Derby were built with JDK versions whose javadoc tool produced HTML files that had a frame injection vulnerability (CVE-2013-1571). The API documentation bundled with this version of Apache Derby has been built with a version of the javadoc tool that does not suffer from the vulnerability.

#### Rationale for Change

The change removes a vulnerability.

#### Application Changes Required

API documentation for old versions of Apache Derby on public-facing web servers should be fixed by

- removing the old versions of the API documentation, or
- replacing the old versions of the API documentation with the new, fixed version in this release, or
- repairing the vulnerable versions of the API documentation using Oracle's Java API Documentation Updater Tool

### Note for DERBY-6224

#### Summary of Change

New SQLPermission needed for JDBC driver deregistration on Java SE 8.

#### Symptoms Seen by Applications Affected by Change

The symptoms are only seen when running an application under a Java security manager on Java SE 8 or later.

When running the Apache Derby embedded driver under a Java security manager on Java SE 8, the embedded driver will not be deregistered during system shutdown unless `SQLPermission("deregisterDriver")` has been granted to `derby.jar`.

If this situation has occurred, the following message will be printed to the Derby error log (`derby.log`) along with a stack trace that tells where the call was made:

`Could not deregister the JDBC driver during engine shutdown. Make sure SQLPermission("deregisterDriver") is granted to derby.jar.`

Also, if an application attempts to deregister a JDBC driver by calling `java.sql.DriverManager.deregisterDriver()` directly, it may fail with an AccessControlException. This is not limited to the Apache Derby embedded driver; it will happen for any JDBC driver, including the Apache Derby network client driver.

#### Incompatibilities with Previous Release

When running on Java SE 7 or earlier, shutting down an embedded Derby engine would also deregister the embedded driver from `java.sql.DriverManager`'s list of JDBC drivers.

When running on Java SE 8 or later, the embedded driver will still be registered after shutdown if `derby.jar` is not allowed to invoke `java.sql.DriverManager.deregisterDriver()`.

#### Rationale for Change

This is a change in the `java.sql.DriverManager` class in Java SE 8. It is not a change in Apache Derby.

#### Application Changes Required

Applications that run under a Java security manager, and that depend on the driver being deregistered as a side-effect of shutting down the embedded engine, need to grant `java.sql.SQLPermission("deregisterDriver")` to `derby.jar`.

Applications that run under a Java security manager, and that call `java.sql.DriverManager.deregisterDriver()` directly, need to grant `java.sql.SQLPermission("deregisterDriver")` to the application code base, and possibly also change the call to `deregisterDriver()` to go through `java.security.AccessController.doPrivileged()`.

### Note for DERBY-6209

#### Summary of Change

When running a server started from the command line with a security manager and a customized policy file, new permissions may be have to be added to it.

#### Symptoms Seen by Applications Affected by Change

Running the server will fail with a AccessControlException indicating a missing permission, whereas running the server in the same way used to work before upgrading Derby.

#### Incompatibilities with Previous Release

Server will not run for affected configurations.

#### Rationale for Change

It is good security practice to limit data base file visibility to the operating system account that creates the database. In this release of Derby this policy is in effect by default when starting the network server from the command line unless this functionality has been explicitly disabled, cf. description of the property `derby.storage.useDefaultFilePermissions` in the Reference Manual. Cf. also Derby issue DERBY-5363.

Note: for other configurations, this policy must be explicitly enabled, e.g when starting the server via the Java API. Similarly, embedded Derby doesn't use the new policy by default. If you enable the new policy in such configurations and a security manager is used, you will need to add the permission described below to the Java security policy file.

#### Application Changes Required

The implementation of this security policy requires that the policy file in use by the server has a few more permissions than before. When running with the default security policy that is taken care of. However, for user customized policy files, these permissions need be added. The permissions that need be added are:

- codebase `derbynet.jar`:

              permission java.lang.RuntimePermission "accessUserInformation";
      	permission java.lang.RuntimePermission "getFileStoreAttributes";
      	permission java.util.PropertyPermission "derby.__serverStartedFromCmdLine", "read, write";

- codebase `derby.jar`

              permission java.lang.RuntimePermission "accessUserInformation";
      	permission java.lang.RuntimePermission "getFileStoreAttributes";

See also the template security policy file provided with Derby at `demo/templates/server.policy` and the section Customizing the Network Server's security policy in the Derby Server and Administration Guide.

### Note for DERBY-6187

#### Summary of Change

Datatype Change in JDBC Metadata

#### Symptoms Seen by Applications Affected by Change

The datatype has changed for two of the columns in the ResultSet returned by DatabaseMetadata.getIndexInfo(). Previously, the CARDINALITY and PAGES columns were INTs. Now they are BIGINTs.

#### Rationale for Change

JDBC 4.2 changed the datatype of these columns.

#### Application Changes Required

Note that on Derby, the contents of these columns is always NULL. Applications may need to be recoded if they expect the metadata for these columns to report that they have INT type rather than BIGINT type.

### Note for DERBY-5866

#### Summary of Change

The meaning of SYS.SYSTRIGGERS.CREATIONTIMESTAMP has changed.

#### Symptoms Seen by Applications Affected by Change

In earlier versions of Apache Derby, the `SYS.SYSTRIGGERS.CREATIONTIMESTAMP` column would tell what time the system clock showed, represented in the local time zone of the database, at the time the trigger was created.

In databases whose format is upgraded to 10.11 or later, this column now represents the time the trigger was created in the UTC time zone. Existing values are not changed in the upgrade, so the creation timestamps for any triggers created before the upgrade will still be represented in the local time zone.

Furthermore, logic has been added to make the values in that column monotonically increasing, so in some special cases (especially if the system clock has been adjusted back in time, or if the database has been moved to a machine whose system clock is behind the clock on the original machine) the column may contain a value that is not equal to the time the trigger was created. The latter change also takes effect if the database format has not been upgraded to 10.11 or later.

#### Rationale for Change

The purpose of the `SYS.SYSTRIGGERS.CREATIONTIMESTAMP` column is to make it possible to fire triggers in the order in which they were created. A timestamp in the local time zone is not ideal for this purpose because there are ambiguities in time zones that observe daylight saving time. Also, there could be ambiguities when moving a database to a machine in another time zone. Storing the timestamp in UTC avoids both of these problems.

Another problem that was seen, was that triggers that were created quickly after each other, could sometimes end up with the exact same creation timestamp. This meant that the execution order for those triggers was non-deterministic. The timestamp of a new trigger is now adjusted so that it is higher than the timestamp of any other existing trigger on the same table. This makes sure the triggers are executed in the order in which they were created.

#### Application Changes Required

Most applications should not be affected by this change in any way. Any applications that read the values stored in `SYS.SYSTRIGGERS.CREATIONTIMESTAMP` should take into account that this column doesn't necessarily tell the exact time the trigger was created, because it contains a value that may have been adjusted in order to ensure the correct execution order.

## Bug Fixes

The following issues are addressed by Derby release 10.10.2.0. These issues are not addressed in the preceding 10.10.1.1 release.

Issue Id | Description
--- | ---
DERBY-6513 | Warnings when building 10.10 branch with Java 8
DERBY-6477 | OutOfMemoryError selecting from SYS.SYSALIASES
DERBY-6463 | localcal demo seems to be obsolete and should be removed
DERBY-6457 | NetworkServerControl API breaks when username or password contains non-ascii characters
DERBY-6456 | Infinite loop in NetworkServerControlImpl when reply >= 32k
DERBY-6455 | Infinite loop in NetworkServerControlImpl.ensureDataInBuffer
DERBY-6443 | ArrayIndexOutOfBoundsException when calling function from trigger
DERBY-6438 | Explicitly grant SocketPermission "listen" in default server policy
DERBY-6426 | Fix isLoginException
DERBY-6411 | Minimal select privilege should be checked in subqueries
DERBY-6408 | EXISTS returns NULL instead of FALSE
DERBY-6396 | NullPointerException in DirFile
DERBY-6387 | Incorrect ordering of triggers in dblook
DERBY-6383 | Update trigger defined on one column fires on update of other columns
DERBY-6374 | Bulk insert of data with nullable UNIQUE constraint fails to detect duplicates
DERBY-6373 | NPE in Statement.getWarnings()
DERBY-6371 | DROP COLUMN looks in wrong schema when checking trigger dependencies
DERBY-6363 | Incorrect evaluation of logical expressions in CASE
DERBY-6361 | Valid statements rejected if Derby has not implicitly created the current user's schema.
DERBY-6357 | CREATE TRIGGER doesn't detect all illegal references to SESSION schema
DERBY-6351 | Syntax error on equal transition table name and correlation name in statement trigger
DERBY-6348 | NPE or assert failure in recursive trigger
DERBY-6346 | NPE in left join whose inner table has a generated column
DERBY-6338 | Remove Profile attribute from jar file manifests
DERBY-6324 | DatabaseMetaData.getJDBCMinorVersion() should return 2 now that we've implemented JDBC 4.2
DERBY-6323 | Fix incorrect @throws and @exception tags in javadoc
DERBY-6320 | Log a page dump to derby.log if ERROR nospc: nospc.U is returned to the user
DERBY-6317 | Optmizer can choose the wrong path when BTreeCostController.java returns an estimate cost and row count of 0.0
DERBY-6314 | Upgrade from 10.10 fails with ClassCastException
DERBY-6297 | Intermittent failures in XplainStatisticsTest.testSimpleXplainOnly
DERBY-6283 | indexStat daemon processing tables over and over even when there are no changes in the tables in soft upgraded database.
DERBY-6273 | NullPointerException when using more than one parameter in COALESCE
DERBY-6270 | Run Java API Documentation Updater Tool on the published javadocs
DERBY-6268 | Run-time statistics not collected if query contains always false predicate
DERBY-6258 | Restrict permissions on BACKUP.HISTORY
DERBY-6224 | Issues with latest JDK 8 EA build because of missing SQLPermission
DERBY-6221 | Can't select from SYS.SYSUSERS if you use a WHERE clause in the query
DERBY-6216 | XPLAIN feature does not work and gives ERROR XCL16: ResultSet not open
DERBY-6212 | NullPointerException when calling a procedure with sub-query as argument
DERBY-6209 | Add release note omitted with 10.9 for new required Security Manager permissions after DERBY-5363
DERBY-6189 | NPE involving temporary table and rollback
DERBY-6187 | Add a release note for metadata datatype changes introduced by JDBC 4.2
DERBY-6185 | Query against view with "where name LIKE 'Col1' ESCAPE '\' " failed
DERBY-6167 | Interrupt restarts clock for login timeout
DERBY-6157 | text for message XSLA4 might encouraging people to delete part of the transaction log
DERBY-6151 | Derby does not report warnings raised by table functions
DERBY-6148 | Wrong sort elimination when using permuted join order
DERBY-6137 | update/delete statement on table with trigger fails randomly with ERROR XSTA2
DERBY-6131 | select from view with "upper" and "in" list throws a ClassCastException
DERBY-6114 | OOME in XAMemTest.testDerby4137_TransactionTimeoutSpecifiedNotExceeded
DERBY-6092 | testPositionAgressive(org.apache.derbyTesting.functionTests.tests.jdbcapi.BlobClob4BlobTest)j fails with : 'The handle is invalid.: java.io.IOException'.
DERBY-6047 | NPE in StaticCallMethod#coerceMethodParameter with a constant argument to INOUT parameter
DERBY-6045 | in list multi-probe by primary key not chosen on tables with >256 rows
DERBY-5979 | ant release target creates a release.properties that has conflicting line endings so automatic checkin fails
DERBY-5886 | FILE_CANNOT_REMOVE_FILE exception prints garbage.
DERBY-5866 | testFiringConstraintOrder(org.apache.derbyTesting.functionTests.tests.lang.TriggerTest)junit.framework.AssertionFailedError: matching triggers need to be fired in order creation:1,NO CASCADE BEFORE,DELETE,ROW
DERBY-5865 | On IBM 1.7 TestDurabilityProperty fails with FAIL -- derby.system.durability=test mode seems to be broken.
DERBY-5823 | Multi-row insert fails on table without generated keys with RETURN_GENERATED_KEYS
DERBY-5610 | ServerPropertiesTest prints .java.net.SocketException: Connection reset to console but test passes
DERBY-5560 | Java deadlock between LogicalConnection40 and ClientXAConnection40 (patch attached)
DERBY-5553 | System property for client tracing -Dderby.client.traceDirectory does not work with XADataSource
DERBY-5416 | SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and then gets mostly garbage collected later on
DERBY-4923 | update of a long row can fail with ERROR nospc: nospc.U exception.
DERBY-4753 | "ERROR 42X01: Syntax error: FALSE." during call to java.sql.DatabaseMetaData.getIndexInfo
DERBY-4729 | add more information to the XACT_PROTOCOL_VIOLATION returned from store.
DERBY-4160 | getMetaData().getIndexInfo crashes with "ERROR X0Y68: Column 'PARAM1' already exists."
DERBY-3940 | Dropping a column does not drop triggers which mention that column
DERBY-2130 | Optimizer performance slowdown from 10.1 to 10.2
DERBY-896 | Cast date to timestamp results in "ERROR 42846: Cannot convert types 'DATE' to 'TIMESTAMP'."

## Fixes in the Documentation

The following issues are addressed by Derby release 10.10.1.4. These issues are not addressed in the preceding 10.10.1.1 release.

Issue Id | Description
--- | ---
DERBY-6473 | Manuals are inconsistent in subsubprotocol information
DERBY-6448 | Document new SocketPermission in Network Server policy file
DERBY-6415 | SYSCS_DIAG.SPACE_TABLE sample query to show all tables and indexes should include schema
DERBY-6403 | Doc for export/import in ref man should mention automatic commit after operation.
DERBY-6399 | clarify backup section in Derby Server and Administration Guide regarding connecting to the backed up db
DERBY-6394 | Fix problems with searchCondition and Boolean expression documentation in Reference Manual
DERBY-6389 | Update adminguide's list of differences between client and embedded
DERBY-6388 | Example WHERE CURRENT OF clause in Reference Guide is nonsense
DERBY-6379 | Manuals are inconsistent in their use of the <shortdesc> element
DERBY-6376 | Revise DatabaseMetaData section of Reference Manual
DERBY-6375 | Suggest use of indexes in devguide's deadlock avoidance topic
DERBY-6369 | Make bullet lists into tables in two Reference Manual topics
DERBY-6367 | Reference Manual incorrectly states that you can't schema-qualify constraint names
DERBY-6355 | Data Integrity guidelines should include not deleting or otherwise tampering with files under the database directory
DERBY-6353 | Update syntax description in Getting Started, use in Reference Manual too
DERBY-6316 | Small typos in devguide examples
DERBY-6298 | Documentation implies column NOT NULL constraint cannot be named, but it can be.
DERBY-6239 | Document optional permission SQLPermission("deregisterDriver")
DERBY-6165 | The shutdown documentation should mention that a shutdown attempt could fail if login timeouts are enabled.
DERBY-6160 | Fixes needed to documentation topics on security policy permissions
DERBY-6129 | Reference Manual gives incorrect datatype for SYSCOLUMNS.COLUMNNAME
DERBY-6127 | Tell users to make sure that the state of a user-defined aggregate needs to be serializable
DERBY-6123 | Optional tools documentation in the reference guide should mention that optional tools require derbytools.jar be in the CLASSPATH
DERBY-6121 | Regularize how we refer to object names in the Reference Manual
DERBY-6120 | Clarify that $ is the separator character which should be used for the external names of static nested classes
DERBY-6116 | Confusing description of limitation on user-defined aggregate names.
DERBY-6103 | Improve documentation of ROW_NUMBER function
DERBY-6090 | Package javadoc publishedapi/jdbc4/index.html out of date
DERBY-5244 | DatabaseMetaData.getColumns(null, null, tableName, null) does not return the columns meta for a SYNONYM
DERBY-4847 | Errors should be displayed if using wrong CLASSPATH to make documentation target for release
DERBY-4629 | Explicitly document how to get case-insensitive sorts and comparisons of string data
DERBY-4628 | The Derby docs would be clearer if we replaced our jargon term "territory" with the term "locale" which is used commonly across the Java ecosystem.
DERBY-4627 | Document how to install your own custom collation for use in sorting and comparing string data values.
DERBY-4280 | Property not documented - derby.language.statementCacheSize

## Build Environment

Derby release 10.10.2.0 was built using the following environment:

- Branch - Source code came from the 10.10 branch.
- Machine - Cygwin on Microsoft Windows 7.
- Ant - Apache Ant version 1.7.0 compiled on June 27 2008.
- Compiler - All classes were compiled by using Oracle the 1.8.0 32-Bit VM (build 25.0-b70, mixed mode).
- JSR 169 - Java ME support was built using libraries from IBMs weme 6.2.

## Verifying Releases

It is essential that you verify the integrity of the downloaded files using the PGP and MD5 signatures. MD5 verification ensures the file was not corrupted during the download process. PGP verification ensures that the file came from a certain person.

The PGP signatures can be verified using PGP or GPG. First download the Apache Derby KEYS as well as the `asc` signature file for the particular distribution. It is important that you get these files from the ultimate trusted source - the main ASF distribution site, rather than from a mirror. Then verify the signatures using ...

    % pgpk -a KEYS
    % pgpv db-derby-X.Y.tar.gz.asc

    or

    % pgp -ka KEYS
    % pgp db-derby-X.Y.tar.gz.asc

    or

    % gpg --import KEYS
    % gpg --verify db-derby-X.Y.tar.gz.asc

To verify the MD5 signature on the files, you need to use a program called `md5` or `md5sum`, which is included in many unix distributions. It is also available as part of GNU Textutils. Windows users can get binary md5 programs from here, here, or here.

We strongly recommend that you verify your downloads with both PGP and MD5.