FOSS4GNA Code Sprint Replacing Vecmath

As Torben indicated on the geoserver blog we got together for a one day sprint after the foss4gna conference. Torben stole my picture for that post so I will have to stick to content ...

Our topic of choice ... replacing vecmath. The use of vecmath library has been a long standing "technical debt" for GeoTools. How long standing? The issue tracker number is GEOT-22.

So what is the problem with vecmath

The vecmath.jar is used by gt-referencing for coordinate system transformations. We only uses a couple classes form the library: primarily to implement GeneralMatrix for use in MathTransforms.

GeneralMatrix extending GMatrix

There is one small problem with this idea - vecmath is not open source! Technically vecmath is distributed as part of Java 3D (an extensions to the Java Virtual Machine). As an extension to Java it was distributed under the same licenses (Sun Binary License and a Research License) as Java.

With the GeoGig project going through LocationTech incubation we have a couple ways to use jars:

• prerequisite: open source jars required to run
• works with: optional jars that extend the functionality if present. These may be proprietary like an oracle database driver.

Although the vecmath license is fine for distribution it does not meet the strictly open source policies required for the GeoGig project. In this case we want GeoTools to include matrix math and needed to shop around for a replacement.

The use of vecmath (as a non open source dependency) also causes trouble for Rich Fecher's proposal to publish GeoTools on Maven Central. 

Enter EJML

With a technical dept page capturing research, some email discussion and a great lunchtime conversation at foss4gna with Eric Engle (the overlap with EclipseCon was good for something) we settled on the recommended Efficient Java Matrix Library (EJML).

GeneralMatrix delegating to DenceMatrix64F

The strategy here is delegate to the DenseMatrix64F implementation provided by EJML, and implement the methods we expect from XMatrix in terms of this new implementation.

The EJML library has the similar arrangement with SimpleMatrix wrapping a DenseMatrix64F an API friendly to casual developers. We were able to use SimpleMatrix as a guide, saving a lot of time.

How to help

While the code sprint was a success in proving the approach, there is a bit of work to go:

• Tyler (from GeoGig) is working on removing the dependency on vecmath (there are a few other Exceptions and data structures in our API that need to be removed).
• Jim (from GeoMesa) wants to write up more test cases to check for regressions between vecmath and EJML
• Although Rich Fecher (from GeoWave) was unavailable to take part in the code sprint - his inspiration to work on this now means we will be hitting him up for a review when the work is complete. Thanks Rich!
• And there is always the question of performance ... will it be faster!

To help take part, or review our work see this branch in github:

• https://github.com/geotools/geotools/tree/matrix

Thanks to Boundless victoria staff, Jim and Andrea for really getting behind this work. This kind of up keep keeps the community ticking along and helps the library be used in more places.

I would also like to thank the new crop of projects using GeoTools for taking part and contributing upstream. It is important to keep our community diverse and your participation is both welcomed and appreciated.

CodeHaus Migration Schedule

As per earlier blog post CodeHaus is shutting down and the GeoTools project is taking steps to migrate our issue tracker and wiki to a new home.

First up I need to thank the Open Source Geospatial Foundation for responding quickly in a productive fashion. The board and Alex Mandel were in position to respond quickly and hire a contractor to work with the system admin committee to capture this content while it is still available.

I should also thank Boundless for providing me time coordinate CodeHaus migration and Andrea for arranging cloud hosting.

Updates:

• Update April 7th: GeoAPI project attachments migrated (using this to estimate time remaining)
• Update April 3rd: Issue tracker signup open to all (no need to email project leads).
• Update April 2nd: Mauro Bartolomeo created GEOT-5074 in the new issue tracker
• Update March 28th: Placeholder tickets created, contents and attachments to follow

Confluence Migration

Is scheduled for ... March 26th! I have taken a copy of the CodeHaus wiki and will be migrating proposals and project history. A html dump of the wiki is published at old.geotools.org so we have a record.

The new wiki is available here: https://github.com/geotools/geotools/wiki

GitHub Wiki

Jira Migration

Jira migration will start on 00:00 UTC Saturday March 28th.

On Saturday all issues will start migrating to their new home (and CodeHaus issue creation will be disabled). If you wish to lend a hand testing please drop by the #osgeo IRC channel on Saturday. Harrison Grundy will be coordinating the proceedings.

We have set up a new JIRA available at osgeo-org.atlassian.net for the migration. You can sign up directly (although we ask you to consider keeping the same user name).

OSGeo Jira

As shown above a few friendly CodeHaus refugees have also been sheltered for the storm (uDig and GeoAPI).

CodeHaus Migration

It with heavy hearts (and a small bit of panic) that we bid adieu to CodeHaus. After many years of service CodeHaus is shutting down. A bit of last moment sponsorship from sonarqube is holding the doors open giving us a chance to migrate in an orderly fashion.

This blog post is a public notice that the GeoTools project will require assistance during this migration due to the short time frames involved.

GeoTools will lock down changes to the issue tracker after the scheduled March 18th release of GeoTools 13.0.

Migration Plan

In this weeks community meeting we have set on a migration strategy.

• Migrate GeoTools issue tracker to a new instance Jira (sorting out host options see below). This is primarily so we can manage large attachments (such as test data) that make our issue tracker valuable. As a fallback plan we will consider migration to github issues.
• Migrate GeoTools wiki to github wiki with a focus on proposals and project history.

Similar arrangements are being made for our friends with the GeoServer (OSGeo), uDig (LocationTech) and GeoAPI (OGC) projects. If you are in position to assist these projects please contact the appropriate development list as they could use a hand.

Due to a technical limitation (the use of a single sign on system) we cannot just take a backup and restore on a new system. This leaves us with two options: Use REST API and a script to migrate to a new system, or assemble a work party to transfer issues via CSV export (and grab the attachments one at a time).

Thus far we have contacted:

• The OSGeo board has offered their support in todays meeting for both setting up hosting and requesting a proposal for data transfer via REST API.
• The OSGeo system administration committee are looking into options for hosting.
  Thanks to both Harrison Grundy and Alex Mandel for helping refine migration requirements on IRC.
• Both GeoSolutions and Boundless have offered assistance
• We have reached out to Atlassian about cloud hosting

Thank you to the above groups for their prompt assistance.

About CodeHaus

CodeHaus has been an amazing resource for our open source community, offering friendly hands on support in the face of the growing pains experienced with SourceForge. We should also thank Atlassian for making their software (Jira and Confluence) available to open source projects on CodeHaus.

GeoTools IP Review Update

There we go, a blog title designed to scare away all but the most dedicated blog readers.

GeoTools Review Status update

After several weeks of email, and a couple days of fixing, I am in the clear to make a GeoTools 9.0-M1 release. Basically the IP review discussion we have been having with LocationTech is complete for GeoTools.

Here are the links:

• https://github.com/geotools/geotools/tree/9.0-M1
• GeoTools 9.0-M1 Release Notes
• GEOT-4578 Assorted IP Feedback and Corrections

Only the subset of GeoTools that is used by uDig was reviewed, and there are still many dependencies left to be contacted. But GeoTools is by far the largest stepping stone.

Thanks to all the project leads who will be (or have already been) contacted. We really appreciate you taking the time to answer a few questions.

Wait Didn't OSGeo do this Already?

GeoTools has gone through the OSGeo Incubation Process, and has an extensive Code Provenance Review.
This review was focused on minimising the risk to the OSGeo Foundation by taking on GeoTools as a project. Some risks the foundation is comfortable with, ie listing all known problems, are not necessarly sufficient for all users (who would like to see known problems fixed).
The OSGeo Foundation does not require projects fix all issues in order to give interested parties a chance to fix (or fund) the work.

Why this is Cool

So why is this cool? The thing is these intellectual property reviews happen all the time, but behind closed doors. If we are really lucky us open source projects get told about the result, or a few private emails with questions.
And if an IP review fails, it fails quietly and we never hear anything again. Indeed if we ask what happened often the organisation conducting the review is forbidden to talk to us about it (for fear of the conversation being taken as advice).
Contrast this with recent developments:

• Google was in position to talk to us about our contributor agreement being unusual. As a result we were able to role out a contribution agreement, OSGeo wide.
• LocationTech IP review is in position to talk with the GeoTools development team, allowing us to answer any questions, and fix any issues that are of concern.
• IBM China was able to ask a few questions via private email which were passed on to the project team as bug reports

These developments lower the barrier of entry for using GeoTools in a corporate setting.

So how did we do?

We kind of nailed it, a big thanks to:

• All those who helped dig into the past
  (ie sort out where the code comes from).
• And Brett Walker for putting together our github pull request policy
  (ie looking into the future and sorting out where our code will come from)

Many project have sorted out how patches work, but have not extended that idea to cover github pull requests.

The only area where we could do a bit better is with the EMF generated code which currently says:

/**

 *

 * 

 */

It would be nice to configure EMF to fill this in next time we generate stuff. As an example for CSW I ended up with:

/**

 * 

 * (C) 2012, Open Source Geospatial Foundation (OSGeo)

 * (C) 2004,2010 Open Geospatial Consortium.

 *

 * This library is free software; you can redistribute it and/or

 * modify it under the terms of the GNU Lesser General Public

 * License as published by the Free Software Foundation;

 * version 2.1 of the License.

 *

 * This library is distributed in the hope that it will be useful,

 * but WITHOUT ANY WARRANTY; without even the implied warranty of

 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU

 * Lesser General Public License for more details.

 *

 * $Id$
 * 

 */

GeoTools FOSS4G 2011

GeoTools had a great showing at the recent FOSS4G Conference (Free and Open Source Software for Geospatial) which was held in Denver last month. Presentations at the conference highlighted how the hard work of the GeoTools developers is paying off in terms of the wide adoption of the library, and its use in some amazing applications.

There were a number of talks directly on GeoTools this year:

• Raster data in GeoServer and GeoTools: Achievements, issues and future developments (GeoSolutions) slideshare
• Status of GeoTools (GeoSolutions,LISAsoft,OpenGeo) slideshare

GeoServer as always provides an excellent example of GeoTools in action:

• The State of GeoServer (GeoSolutions,OpenGeo) slideshare
• GeoServer on steroids (GeoSolutions)
• The status of GeoServer WPS (GeoSolutions)
• Advanced cartographic map rendering in GeoServer (GeoSolutions)
• Web Mapping with GeoServer (OpenGeo)
• Practical use of GeoServer & SLD for WMS (GISassistANS AB)
• Integration of GeoServer with NoSQL databases (Data Tactics Corp)
• Mapnik2GeoTools (OpenGeo)

These presentations are a great way to see what is possible with GeoTools. Often it is easier to understand the capabilities of the library by seeing it in use, and both the State of GeoServer and Advanced cartographic map rendering presentations are excellent examples of this.

A special mention to GeoScript which is building a bridge between GeoTools functionality and a range of scripting languages, with the emphasis on providing a concise and simple API:

• GeoScript - Spatial Capabilities for Scripting Languages GeoScript.ppt slideshare(OpenGeo)
• Working with GeoScript (OpenGeo)
• Scripting GeoServer with GeoScript (OpenGeo) slideshare

Geomajas had a strong showing this year with several presentations plus a hands-on workshop:

• Introduction to Geomajas (Geosparc)
• Geomajas, a GIS framework for the web(Geosparc)
• Mobile GIS through Geomajas(Geosparc)
• Securing GIS data(Geosparc)

We are hoping to attract more direct participation of Geomajas developers in the GeoTools project (for now we just have a welcome page).

Did I miss you project in this roll call? Contribute a welcome page to the GeoTools User Guide in order to be included in promotional activities!

GeoTools Codesprint

Finally a big thanks to those taking part in the GeoTools code sprint!

• Looked into why sorting is turned off for MemoryDataStore!
• Put together a Process Tutorial (OctagonalEnvelope in 5 lines!)
• A couple great volunteers (shown below) updated the Eclipse Quickstart to work with the included copy of Maven
• Looked into Neo4j - but got stuck on AGPL license (grrr!)

Current work on the gt-swing module

The gt-swing module, which provides GUI widgets for GeoTools applications, is in transition. This is a technical term for broken - but broken with a mission: gt-swing module is being overhauled to become a fully supported GeoTools module.

What does this mean ?

Well, right now it means that if you want to run any of our tutorial applications you will need to use GeoTools version 8-SNAPSHOT. This is the development version which is updated daily. We wouldn't normally inflict this on those new to the library, but at the moment it is the only way you will get the tutorials to work.

What else does it mean ?

A better gt-swing module with lots of new features, better documentation and easier to understand code. In particular, the JMapPane and JMapFrame classes have been largely re-written and there is a new JLayeredMapPane class to make it easier to do animations and custom graphics over a map display.

When ?

Real Soon Now. Much of the work has already been completed. See the GeoTools issues reporter for a summary.

GeoTools has been so good to me I really want to help: what can I do ?

To date, we've only had enough people-power to work on the trunk sources in the GeoTools code repository (version 8-SNAPSHOT). We are looking for volunteers to help get the gt-swing module working again on the GeoTools stable branch (version 2.7). If you can help, please sing out on the user list. We will like you a lot.