GeoTools

Wednesday, January 2, 2019

GeoTools Java 11 Code Sprint Results

With the recent GeoTools 21-M0 Milestone release available for everyone to try out ... we can finally report back on our success at the OSGeo Java 2018 Code Sprint.

References:

Java 11 Quickstart shows how to use the GeoTools with the new module system
Java is still free for background on Java 8 and Java 11 support options (or our Java Install page summary table)

Technical Challenge

The secret to a successful code-sprint of this magnitude is preparation and communication. Once we started to get a handle on the challenges involved (thanks to Torben, Andrea, and Jody for initial research) we were able to pull together a plan to work together.

A key technical challenge required repackaging the GeoTools library, an activity we wished to do in conjunction participating projects. Easier to do changes one at time, testing in downstream projects, to catch any issues early.

The sprint broke activities down into several stages.

Stage 1: Build and run in JDK 11
Stage 2: Reduce warnings from dependency analysis tools
Stage 3: Resolve spit-module warnings (and GeoTools Quickstart application can run on the module path)

Early Stage 1 Build and run on JDK 11 results

Stage 1 and Stage 2 got off to a running start with Andrea and Brad spending a lot of time before the sprint setting up java11 branches on all the projects (and almost getting all the builds to go.) Indeed they were well on the way into stage 2 using the automated tools to get lists of dependencies for the team to review.

When updating your own projects:

jdeps included in JDK provides static analysis of jars identifying potential issues
jsplitpkgscan scans jars looking for
running tests and looking for warnings (only way to catch invalid reflection use)

An example of jdeps output:

jdeps --jdk-internals -R *.jar
ehcache-2.10.3.jar -> jdk.unsupported
net.sf.ehcache.pool.sizeof.UnsafeSizeOf -> sun.misc.Unsafe JDK internal API (jdk.unsupported)
gt-coverage-api-21-SNAPSHOT.jar -> java.desktop
org.geotools.coverage.io.util.Utilities -> sun.awt.OSInfo JDK internal API (java.desktop)
org.geotools.coverage.io.util.Utilities -> sun.awt.OSInfo$OSType JDK internal API (java.desktop)

While there were many difficult stage 3 tasks, a quick thanks for Ian Turton updating the version of EMF library used for all our xml parsing/encoding data structures. This is the kind of careful work that when done well nothing breaks, and nobody notices.

What is new

GeoTools is now compatible with Java 11.

All GeoTools modules have an assigned automatic module name and can be used in a Java 11 Jigsaw modular application. The module name is typically the name of the topmost package in the GeoTools module - for example, gt-main has the name org.geotools.main.

For a tutorial on how to use GeoTools modules in Java 11, refer to the Java 11 quickstart.

Note: Our codebase uses target=1.8 for compiling, however if you are building with Java 11 the result will only be usable with Java 11 applications (due to a subtle library change where the system libraries change type narrow the instance they return, the new compiler picks up this making the result unusable on Java 8). We recommend building with Java 8, as the result will run can be used on with both Java 11 and Java 8.

What has changed and how to upgrade

In addition to the usual changes (grabbing newer versions of dependencies) we have had to make some significant changes repackaging the GeoTools library itself:

The gt-api jar is no longer provided, the classes formally contained here have been distributed between gt-referencing, gt-metadata and gt-main as appropriate.
gt-metadata had several packages changed examples include:

Factory moved to org.geotools.util.factory.Factory
Hints moved to org.geotools.util.factory.Hints
GeoTools moved to org.geotools.util.factory.GeoTools

gt-referencing examples:

CRSUtilities moved to org.geotools.referencing.util.CRSUtilities

gt-main examples:

SLDParser moved to org.geotools.xml.styling.SLDParser
Many packages were moved to gt-data!

gt-coverage examples:

CoverageUtilities moved to org.geotools.coverage.util.CoverageUtilities

gt-xml examples:

GTXML moved to org.geotools.wfs.gtxml.GTXML

gt-appschema examples:

Many classes moved to gt-complex

As shown above many of the conflicts involved small utility classes so we hope the impact to downstream code will be minimal. Our most deliberated change was deciding which import to break CommonFactoryFinder or Hints. We found slightly more references to CommonFactoryFinder so it remained unchanged!

Our priority for this work was leaving class names unchanged allowing your IDE to quickly correct:

IntelliJ delete your import line and use auto-complete the import (Alt-Enter)
Eclipse provides the organize imports command (Control-Shift-O) to fix all imports when selecting a file or folder
The complete list of changes is on the Stage3 tab of our spreadsheet for any parties wishing to turn this into a search and replace script

What we were unable to address

There is a list of items we were not in position to replace (or warnings we were unable to address):

Java Advanced Imaging (jai_codec, jai_core and jai_imageio) depend on java removed internal api. We will need to look at options in 2019.
Java Advanced Imaging and ImageIO native implementations were not expected to function in Java 11 as the extension mechanism they used is not longer provided (this is documented in our user guide.) We will continue to use the pure-java implementation of these libraries, indeed improvements in java compilers have gradually eroded the performance advantage of these native libraries.
Some functionality (ehcache, and gt-geobuf) depends on jdk.internals access. We will upgrade to new versions of these libraries will be made available.
At runtime a few libraries (hsqldb, parboiled) make use of illegal reflective access. This results in some unavoidable warnings in the logs but does not prevent applications from running.
We did not hunt down replacements for various testing libraries many of which depend on reflection to get their job done.
We looked at dropping gt-opengis as a stretch goal for "stage 4" but had more than enough work to do.

Looking ahead

GeoTools is supporting both Java 8 and Java 11 giving our community a chance to migrate their work to Java 11 when ready.

This has been quite a change of pace for the Java ecosystem:

With Java 11 our community is now lead by an open source project, a change we wholeheartedly support.
Java 8 continues to be supported we free updates, which is an unexpected and welcome surprise that speaks well for the health of the platform as a whole. Seeing a range of vendors and commercial interests collaborate in a responsible fashion is heartening.
The Java platform has a lot more freedom to grow and innovate, with some commercial motivation for improvement. After so many years of stability this will take some getting use to for our development and operational community.

For the GeoTools community we have met this initial challenge and can see where additional work will be needed in the months and years ahead. Thank you for your continued support and happy mapping.

Java 2018 Code Sprint

The code sprint was distributed this year with groups meeting in Europe and North America or remotely coordinating online. The funding priority for this sprint was helping developers with travel costs so we could get as many people together to work together in person as possible.

We would like to thank OSGeo for hosting the event, and our sponsors Gaia3D, Astun Technology, atol, and OSGeo:UK.

Showing up to the Party

World wide participants came from a range of organizations or helped out individually. Participating organizations are considered as in-kind support and are deserving our thanks:

In the Boundless Victoria office we arranged perfect gray weather sprinting, with a small bit of sunshine for fish and chips along the docks. A big thanks to James from CCRi and David from Boundless who made the long flight over.