gvSIG CE using GDAL 1.9.2 to read raster data, including ECW, JP2, MrSID and Lidar files

Recently, I've been working for gvSIG CE project, thanks to the folks of geomati.co (I'm part of the team now!). I was solving some GDAL related open cases (this, this and this). the aim is to get a functional and stable gvSIG CE 1.0 version. There are still a couple of cases to solve (this and this), but I think we're pretty close. I'd like to explain what I did to solve these cases.

Make gvSIG CE work with GDAL 1.9.x

This bug was hard to find but easy to fix.  First, a 1-minute introduction to GDAL use in gvSIG CE. gvSIG CE uses a JNI wrapper (based on GDAL 1.7.x GDAL packed JNI wrapper) to make C++ native calls to GDAL library. There are 2 important GDAL JNI wrapper classes involved in most GDAL operations:

Gdal.java (it interacts with native C++ GDALDataset class)

GdalRasterBand,java (it interacts with native C++ GDALRasterBand class)

Both classes inherit from the same base class: JNIBase. This class defines a function named baseSimpleFunctions that calls one of several native C functions, depending on an input parameter. But some of these native C functions are GDALDataset-based, and other are GDALRasterBand-based. I mean, the pointer passed to them doesn't hold the same C structure (wrapping C++ object) in all cases.

And here was the thing. To actually read the raster data, the Java function GdalRasterBand::readRaster is called. This function checks the position and size of the region being accessed, comparing it with the raster dimensions. To check this, the C functions getRasterXSize and getRasterYSize are called. These functions expect a pointer to a GDALDataset C object, but the GdalRasterBand Java class provides a pointer to a GDALRasterBand C object. I had to replace those Dataset-based calls to RasterBand based calls, to match the underlying C structure.

Replace ECW and MrSID driver with GDAL

First, a few comments on these formats. ECW stands for enhanced compression wavelet. This propietary format is a high-performance image-compression format designed specifically for geospatial imagery. It's patented and owned by Intergraph. More information here.

Intergraph provides a license needed SDK for encoding data and a free SDK for reading. Since version 4.1, this free SDK is only available for Desktop development on Windows systems. There are promises to include Linux support since 5.0 version, but gvSIG CE doesn't plan to support ECW and other proprietary formats. So, I worked with the last known version working on Linux systems too. This is 3.3.

About MrSID, it stands for multiresolution seamless image database. It's a proprietary format too, owned by LizardTech. As happens with ECW, there's a free SDK to read MrSID files. It can be downloaded from here (MrSID SDK, under Tools & Utilities. Free register required). And as happens with ECW too, gvSIG CE won't directly support this format. So, it delegates in GDAL.

In both cases, we'll need to compile GDAL with the proper support. So, I had to download and compile MrSID and ECW libraries and build GDAL using --with-ecw and --with-mrsid options.

As there is a JNI wrapper for GDAL, there are JNI wrappers for ECW and MrSID. The goal was: get rid of them, and give GDAL credentials to handle these formats. Four steps involved here:

- Compile GDAL with ECW and MrSID support, as said

- Tell GDAL to take care of those formats for reading (class GdalDriver)

- Unregister ECW and MrSID drivers from gvSIG CE raster library (libRaster)

- Delete ECW and MrSID wrappers and native code.

Once done, you have gvSIG CE using GDAL 1.9.2 to read raster data, including ECW, JP2, MrSID and Lidar files (previously managed by ECW and MrSID drivers).

Jorge Arevalo

SEXTANTE news and inofficial preview

Dear All,
As some of you may already be aware, there have been substantial changes to the SEXTANTE project.
The website at www.sextantegis.com now focuses on a new implementation of SEXTANTE, written in Python and designed primarily to run from within QGIS.
In response to this, we have contacted Victor Olaya, the original creator of SEXTANTE, and he agreed for us to transfer the Java source code to the gvSIG CE SVN. From now on, we will keep developing SEXTANTE as an official part of the gvSIG CE project.

Over the past few weeks, I have made extensive changes to the SEXTANTE settings system and the graphical Modeler. You can download a snapshot of the current SEXTANTE version here
Copy the ZIP into "../extensiones/es.unex.sextante" and unpack it, overwriting any existing files. You might also want to delete your old settings in the "Sextante" folder in your home directory.

The most important improvements are:
1. Renovated settings pages.
2. Better portability: GRASS, SAGA and R are by default looked for in subfolders of the SEXTANTE installation folder.
3. Brought back the ability to set up a GRASS installation anywhere on your system as an algorithm provider.
4. Vastly improved graphical modeler. This tool is now stable and should be fit for productive work.
5. GRASS vector modules work on 64 bit Windows.

For now, you will also need to select "High compatibility mode" on the GRASS settings page. This will make sure that SEXTANTE runs with any version of GRASS 6.4.x.
I would especially like your feedback on the new graphical modeler (note that the documentation is currently out of date). If you find any problems, please report them via the bug tracker (don't forget to select "SEXTANTE" as current project first).

Best,
Benjamin Ducke
gvSIG CE Team

Running gvtools from sources

After some work we are able to share some results, basically a methodology [1] that can be "described" and followed by more than one person. We have created a new empty repository where user interface functionality is added little by little from the old codebase and the old non-GUI calls are replaced to calls to a new, in-progress Geotools-based library.

Some of these functionalities are very easy to include, some require changes to the GT-based library and take more time. In any case, now we can estimate roughly the cost of the integration since we have been tracking the time spent and we know how many functionalities are still to migrate. Anyway, we'll see how many people is interested on it enough to give us a hand and share the effort.

We have created a project on github that contains the work in progress [2]. As it can be seen in the developer guide [3], a "git clone" and a "mvn package" is enough to make it ready to run. You can find some general description of the project here [4].

Best regards!

[1] http://gvsigce.blogspot.de/2012/11/geotools-integration-methodology.html
[2] https://github.com/gvtools/gvtools
[3] https://github.com/gvtools/gvtools/blob/master/doc/source/introduction.rst
[4] https://github.com/gvtools/gvtools/blob/master/doc/source/developer_guide.rst

Geotools integration methodology

The Munich codesprint was some time ago and we did not have the opportunity to devote more time to the Geotools integration since then. Finally we have been able to work on it for a week.
As stated from the start, the aim of the 2012 codesprint in Munich was not to finish the Geotools integration but to get an idea of how to tackle the task and estimate the cost of it (see our Release Planning for details on how we want to tie this into the CE release cycle).
First of all, for all that have not dealt with the gvSIG code base before: We have found that this task is not easy at all. The gvSIG 1.x branch (the one CE is based on) has hundreds of thousands of lines of code, organized in ~50 different projects that not only do not completely separate user interface from model but that are also tightly coupled with each other. Little or no documentation, duplicated code, meaningless variable names and 17.000 warnings makes handling the gvSIG source code a really discouraging experience. If you're a developer, I'm sure you get the idea. 
After trying different approaches and restarting from scratch each time, we have finally found a method that has the following advantages:

• It provides value immediately. We'll be able to produce gvSIG versions working on Geotools very soon. Of course, the initial versions will lack a lot of functionality the current gvSIG CE version has, but:

1. everybody will be able to see the progress of the integration
2. at a certain point in time, some people may be content with the status of the integration and start using it

• It defines several more or less precise steps to follow. This is helpful when dealing with a beast of so many lines of code. The method is explained further down.

• Can be done in parallel by several people. This is very important since this initiative has received some attention and I would say it is likely that someone will join and share the efforts. The development will take place in a GIT repository in github, so that forks and merges can happen easily. 

This method has been the first one we have considered good enough to make the integration succeed. Of course, suggestions are welcome.
Let's explain the method. In gvSIG, there is a plugin system, called "andami" that hosts about 25 extensions that provide the user interface. These extensions use several libraries to implement their functionalities. Instead of modifying the existing code base, we have created one more extension project called "main" and one more library project called "core".
The main extension will receive one by one every functionality from the old codebase. As the libraries used by the old code are not present in main, there will be some compilation errors. To solve these compilation errors, the code that uses the old libraries will be changed to use the core library, adding the necessary functionality to core if necessary. Of course, core exposes Geotools API, as it was the main aim of the integration. 
So schematically:

• Move all extensions to main one by one

• For each extension:

• Include all classes in the same extension that are necessary to compile. Code can be removed if:

• Supports old 0.3 version (method name ends with 03)

• It is not necessary right now and we'll be forced to add it later. For example:

• Public methods that are not called in main -> Either they will be called with code that will be moved to main and we'll recover the erased method, or they will never be called so it was well removed.

• Whole classes -> At the end we can compare old and new codebase and we can detect the classes that are missing.

• Fix the compilation errors of code using old libraries (not accessible anymore from main). This might require adapting the code to the new core library and even adding more functionality to it. In case of new functionality:

• Discuss in the list the best way to implement

• Implement

• Add javadoc

• Add unit tests 

Some considerations when following the method:

• Our first priority is integrating Geotools. Please don't try to fix the extension code in main. While following the previous method you will see some things that can easily be fixed. Please don't. You'll realize that there are many of these and fixing them will keep you from the real aim, which is the integration. Let's try to do only minimal changes to the code that goes to main.

• Do not add TODOs. They are ignored. Assert false:"todo message"; is much more effective. If you don't want to implement a method, try to remove it (following the rules stated before).

• If you add some comment related to the integration, include the text "gtintegration" in it, so that we can process it later.

• Please, try to understand core before making changes. Just use the mailing list to agree on some changes before starting to code. 

gvSIG CE code sprint summary

The first gvSIG CE Code sprint took place from Monday October 15th to Friday October 19th 2012 in Munich. 100% of the cost of this code sprint have been covered by ArchIV, Benjamin Ducke, CSGIS, Cartolab and Geomati.co. It has been a very intensive and productive week with these participants:

Benjamin Ducke <benducke[at]fastmail.fm>
Fernando González <fernando.gonzalez[at]geomati.co>
Ruth Schönbuchner <ruth.schoenbuchner[at]csgis.de>
Johannes Valenta <office[at]arch-iv.de>
Jose Canalejo <jose.canalejo[at]csgis.de>
Víctor González <victor.gonzalez[at]geomati.co>
Francisco Puga <fpuga[at]cartolab.es>

Benjamin Ducke worked on the integration of GRASS GIS in SEXTANTE 1.0 and managed the gvSIG CE release plan. The next release will be a beta release for gvSIG CE 1.0. We plan to release the first beta version of gvSIG CE 1.0 in November 2012 and the final version 1.0 before the end of that year. It will feature many bug fixes, user interface improvements, and some new/updated extensions (most importantly, a new version of SEXTANTE, OpenCAD and NavTable).

Fernando and Victor Gonzalez were examining a possible Integration of Geotools in gvSIG CE. This is a very important task for gvSIG CE because it will improve hugely the development of gvSIG CE. At this point we would like to say a big thank you! They have invested several days on this work without any sponsor. The first results allow us to estimate a roadmap and evaluate next steps. Surely more investigation and testing will be needed.

Francisco Puga helped us accomplishing the best integration of OpenCadTools and Navtable in gvSIG CE and he also discussed the integration of geotools into the official gvSIG as this task could be also important for the gvSIG development.

Ruth Schönbuchner, Johannes Valenta and José Canalejo were working on updating the project documentation. The website has been updated, a gvSIG CE Quickstart based on OADE Quickstart has been put online. An example project with sample data will be part of the next CE 1.0.

The code sprint is a great opportunity to make decisions about the future of the project. It has been a very good experience for all of us. We are looking forward to the next code sprint! If you like to discuss next steps of the project we would like to hear from you (e.g. add a location for the next code sprint: Toulouse, Girona, Berlin, ?). Join our community list or follow us at twitter.

We're grateful to all the developers for their awesome engagement! See you soon and thank you very much to all participants!

The gvSIG CE Team

Integrating GeoTools into gvSIG CE

Fernando and Victor Gonzalez (http://geomati.co/) are working on a possible integration of GeoTools into gvSIG CE.
We will surely keep on discussing this important task at the next gvSIG CE code sprint in Munich.
Congratulations and thank you very much!

New videos on gvsig ce youtube channel

New videos regarding network analysis were uploaded to gvsig ce youtube channel:
http://www.youtube.com/user/gvSIGCE

The videos refer to the following three topics:

• how to generate a network
• how to perform shortest path analysis
• how to exectute origin-destination matrix

Other functions (Closest facility, Connectivity, Minimum spanning tree) want to be tested and videos will be done in parallel.

Have fun! Ruth