1 2 3 Previous Next

jive

34 posts
jive

Note to Self Blog

Posted by jive May 24, 2012
One of the Open Source Geospatial Foundation (OSGeo) email lists is reviewing relationship between OS and IP. Again. I am a big fan of matching license to community. While I recognise that open source and intellectual property can be considered as opposites - I find myself lumping them both together. Rather then recycling this viewpoint, here is a note to self (allowing me to supply a link): 

Note to Self: Open Source / IP

Open source is a part of the intellectual property conversation. Open Source and IP deliberately go together; indeed it is what the license part of open source is about. Open Source provide a range of licenses which provide guidance around the use (and reuse) of code. These licenses facilitate, by defining the terms under which, which we can reuse / recycle code with the permissions of the authors (i.e. copyright holders). Since OSGeo is an open source community we often have direct contact with these authors; in a few cases they have signed over copyright to a neutral organisation - such as the OSGeo foundation itself. It would be nice if software worked like the fashion industry and did not make use of copyright protection. Since our industry does make use of these provisions - it is great that open source allows us an "out". The specific nature of the open source license used by a project lets you know what has been negotiated with the authors with respect to IP. Depending the different aspects of the IP spectrum you agree with you may react strongly (or indifferently) to the options available. You can even use this as a measure of what you care about (either from the case of trying to protect; or from the case of trying to access). For myself I try protect the freedom of developers to get the job done. Open source is one of my best tools to protect this freedom. Another valuable tool is the different foundations. You can tell a lot from the a foundation and its members by what makes their hit list when accepting new projects. OSGeo is very much about transparency; but not so much on the protecting developers from IP issues. Eclipse foundation suffers on transparency a bit; but is very responsive to protecting developers from IP issues. Aside: The uDig community is looking at joining the Eclipse "location working group". It is a good fit of technology, open source spatial outreach, and access to a pool of developers we do not normally interact with. Perhaps when this process is done (or even started) we can provide further feedback on the experience.  
jive

ObjectAid Blog

Posted by jive Apr 11, 2011

ObjectAID is the first UML tool you may actually use; I know I should talk about its features or something - but the real reason you may actually use this application is that it is Fast.

1. Run the wizard to create a new diagram

2. Drag and Drop your classes into the diagram

3. I have found the "auto image" setting helpful so it writes out an image (PNG,GIF,etc..) every time you make a change.

Here is one of the many diagrams pulled together for the GeoTools User Guide I have been rewriting.

GeoTools FeatureSource

You can install ObjectAID from the eclipse marketplace (in the help menu).

jive

FindBugz Community Review Blog

Posted by jive Mar 13, 2010

I am very fond of FindBugz (indeed it has found its way into the QA process of most projects I work on....). When visiting the site to check Eclispe 3.5.2 compatibility I found they were working on a new tool.

Indeed a very interesting tool.

Looks like they have a specific tool for browsing the output of a findbugz; and sifting through the results in order to mark each issue. They have an example for Eclipse 3.5 listed.

Now I think I get this result when I run FindBugz direct in eclipse (it produces a bunch of warnings for me to sift through). What is interesting is the Classify and First seen fields. It is my hope that this tool can be hooked into the bug tracker for each project allowing people to raise an issue directly for review.

jive

How 2 Map Blog

Posted by jive Apr 8, 2008
I will be using this blog a little less; it seems the vast majority of my postings are of interest to those involved in open source spatial (and Java is just the canvas I enjoy working with). As such I will be making use of the following blog: http://how2map.blogspot.com/. I would like to thank Java.net for hosting, and hope the spam comments problem abates.  
An open source mailing list I am on recently took up the interesting debate about the Merits of Maven going on in the blogosphere. Occasionally I remember I have a blog; and most of my email is too long anyways ... 

Frank Hardisty asked: For projects the size and complexity of GeoTools and GeoServer, is there currently a viable alternative to Maven?

Is there an alternative to Maven

Good question - I think the alternative would be more organization on our part:

  • less dependencies
  • and to commit the jars into svn
There are two downsides: 
  • writing a lot more of our own software; and end up in a not invented here situation like many other projects.
  • at the end of the day all we would be left with is a simple build
Personally I would rather share the burden of making a build system with other projects; ant is at the same primitive level as make. Scrips are produced and copied from project to project; and after a while it gets so fragile everyone is scared to touch it. 

What I like about maven is the chance to look into some coverage tools (for example), notice that they support maven, and quickly try out a few by running maven a couple of times. I have to hand it to people like Justin and Martin who have gone ahead and made a few maven plug-ins for the community; I am so glad our build is not a mess of ant scripts, and hacks that only work on linux, and ... well lets say I have worked on a smaller project that took 40 mins to build.

On a related note I am stuck maintaining a simple build that does the exact same dependency management work (since it uses GeoTools jars) - the uDig application uses an ant script to suck down jar files from the maven repository. Maintaining this is a pain and we really wish we could make use of maven.

The uDig project is stuck between two evils; something called PDE Build which makes maven look like a cake walk; and Maven (who would solve our problems except that the eclipse foundation is a big pay to play mess).

Jody

PS. My rant here ends up being pro community; rather than pro maven

PPS. A cake walk is actually an American dance tradition; with the best dancer literally "taking the cake"

PPPS. There is a couple of Java Community Process module proposals going around trying to learn from both Spring, OSGi and the maven repository system

One of the advantages of being a library is that we can really be everywhere. The downside is that even if you attended the conference (wasn't it great!) you could not of managed to catch all that we had going on. I have written up an article over on the GeoTools wiki: GeoTools at FOSS4G Article. For those that wanted to see how the MapServer vs GeoServer performance testing turned out follow the link and have a look at the pdf. My favourite development this year was represented by two tag team presentations on twisting Java Advanced Imaging into some simply amazing raster work (Next generation of raster support for the GeoTools-GeoServer stack) including a combination PostGIS / File system combo for x,y,z,t rasters (Managing WMS and WCS multidimensional NetCDF Datasets with Geoserver).http://docs.codehaus.org/download/thumbnails/14876688/ThisTruckIsSlow.jpg I am going to talk about about the two Labs I was involved in. Labs are a great feature of the FOSS4G experience. So often at these conferences all you get it talk, talk and more talk. Labs are the opposite of that - hands on time with real software. 

How to Cope with GeoSpatial

This was the only hard-core hands on programming lab at the conference this year. Thanks to everyone who attended! GeoTools was used to access a Web Map Server, generate a Shape file, hack away at PostGIS (using Common Query Language and Filter 1.0). Students that finished early got a chance to visualization with Images and Shapefiles. Thanks to Martin, Brock an Melisa for pulling this one together. Special thanks to Andrea and Jesse for the code review If anyone wants to see more GeoTools documentation please hire me for a training course ;-) 

An Introduction to the uDig Open Source Desktop

This lab was my highlight of FOSS4G this year. I have been teaching the uDig training course for several years now - but it is focused on getting development teams up and going. This is was my first time showing uDig to a room of users - and it was great to see how happy it made everyone. I had originally planned this course as an intro workshop (so people could hunt down he projects they thought were fun over the course of the week). The fates of scheduling were against me - this was the last Lab on the last day. On the bright side everyone had something they wanted to "see in action". If you would like to try this workshop at home it is available online: We had several members of the uDig development team in attendance for some one-on-one help (mostly they stood around with big grins on their faces). At one point a circle of mac laptops gathered like a fairy ring at the back of the room as a flash drive was passed around. 

What is Next?

http://docs.codehaus.org/download/thumbnails/14876688/Fun.jpgSilvia (above) used the FOSS4G code sprint to create a brand new Italian translation. Thank you! I would like to ask for additional languages - if you are able to translate please stop by the download the latest let me know how well it works in your organization.  
The Free and Open Source Software for Geospatial conference is ramping up and Java should be everywhere. I have the privileged of working with a number of people doing presentations this year - and it has been great watching the GeoServer project (running Java EE) catch up to the long time king of the hill MapServer (written in C). 

If you want to see how close these two are be sure to attend the presentation: WMS Performacne: Mapserver vs. Geoserver.

It is a good thing I got a sneak peek at the results, because I am going to be busy ...here are a couple of things to see out of the way of the main program.

Demo Theatre

This year at FOSS4G we have a full on Demo Theatre set up; think of these as a lightning talk combined with actual running software. I can think of few better ways to get a feel for what is going on then sitting back and watching these programs actually run. The Demo Theatre is happening during the coffee and lunch breaks; and there is a speak Google Summer of Code slot Wednesday Morning where you can take a look at what the next generation of developers are working on. 

Developer Gatherings

If you want to hunt me down and talk shop please visit the Refractions booth. We will have a live PostGIS and uDig demos going as part of the integration showcase. The other place I will at least visit is the OSGeo booth, with OSGEO formally running the show this year we may have the place to ourself? 

There are two places to hunt down the developer community and see them in action:

See you there!  
JSR-275 is almost ready! And there is much rejoicing to be had - but not quite yet.  In my last blog post I talked about GeoAPI and how we use it to define interfaces for a lot of the spatial goodness happening in the Java world.  One of the constructs discussed was the interface for CoordinateReferenceSystem. This interface is implemented by a couple of projects:Central to the idea of a CoordianteReferenceSystem is what unit of measure the numbers are in (meters, feet, sexidecimal degrees, etc...).  JSR-275 defines a really amazing concept of "Unit of Measure" (as described in Fowlers Analysis Patterns Book). This idea is powerful, and allows you all kinds of runtime safety checks so that you do not (in the proverbially case) mix up meters and feet and fly into Mars by accident.  In JSR-275 they broke out Java Generics system and made a lot of this goodness into something the compiler can check for.

So why doesn't GeoTools use JSR-275

Problem is the reference implementation (also provided by JScience) is Java 5 only :-) So for the moment GeoTools community is hanging back so we can work with all those Java 1.4 J2EE projects that pay the bills.

JSR-108 Where art thou

Actually the GeoTools project still uses a units-0.1.jar provided by the earlier JSR-108. This earlier JSR-108 was withdrawn - and that seems to have scared the voting parties.  From the JSR #275 Units Specification JSR Review Ballot:On 2005-06-09 Sun Microsystems, Inc. voted No with the following comment: Sun is voting "no" on this JSR based on concern over whether there is broad need for this JSR and concern that this is essentially equivalent to an earlier JSR (108) which was previously withdrawn.  It is possible that this topic may eventually benefit from a JSR. However, it is not yet clear that there is broad Java community interest in a Java standard in this space.   There appeared to be only fairly limited community interest in JSR-108.  I would like to suggest that the submitters continue investigating and developing their API as a standalone class library.  Once they have more experience and can demonstrate broader community interest, it may be appropriate to launch a JSR.                                                           - Graham There are a lot of very similar comments, even the positive votes call into question the existence of JSR-275 based on the perception of JSR-108 as a failure.

Why Units Matter

Please be assured that the concept of a unit system is vital to using Java for scientific purposes. Even the old JSR-108 code is in day to day use around the world - in my projects it is used to explain how the world is round.  Be assured JSR-275 would be an instant success if it was not battling a community that is still bound by Java 1.4. The availability of JSR-275 will be added motivation for us to upgrade. The value and utility of JSR-108 is also holding us back; be assured that when we do upgrade the reference implementation of JSR-275 will be test from all kinds of angles.

GeoTools and Java 5 - now is the Time

GeoTools is tabling a proposal right now about when to upgrade to Java 5. The success of JSR-275 on July 8th will be instrumental in this decision.  We will do our best to upgrade to JSR-275 - The only other thing you can do to help is is to kick websphere (and others) into releasing a proper Java EE implementation. You will find that the scientific community, much like the rest of Java, is splintered between versions.  Units with generics is great, it makes for a better example of Generics pulling their weight in the Java language than the usual example of Collections.
Someone pointed me at an interesting new project .... actually an interesting old project in new cloths. The Spatiotemporal Epidemiological Modeler is a little component for the eclipse Open Healthcare Framework. And it includes one feature (out of many) of interest to this discussion: A GIS. 

Perfect!

This is exactly what I want to see happen; GIS as a widget that can be rolled into your application de jour. The actual widget is kind of fun; the data format makes use of Geography Markup Language (GML) to define its polygons and so on. Yes it is an Eclipse RCP application, and no they did not talk to us about using uDig. You can uDig in your own RCP applications - but what they have is nice and specific to their data format. They even have google earth integration - looks like they could of talked to GeoServer and saved some effort. I assume there was just not enough time / energy / benefit to the developers to work with a larger community on this one. uDig offers an intergration platform so you can roll all kinds of data formats into the same projection and see it on the screen - making use of a number of open source projects to get the job done. 

What do we have to work on?

There are several areas of neutral grounds for Java open source projects to meet, greet, and share common geospatial interfaces. At a low technical level most projects are making use of the JTS Topology Suite to get their polygon game on. Sometimes they wrap the beast up in modern ISO Geometry clothing, and sometimes they just punt raw coordinates onto the screen until JTS is needed for actual work. At a high level we are often working against the same file formats (shapefile, JPEG2000, etc..) and against the same web services (Web Feature Server, Web Map Sever). It is in between where things get a bit messy. Geospatial projects often use the same concepts (Feature is something that can be drawn on a Map, A Map is made up of layers of features styled up and made to look pretty, etc...). Occasionally these concepts are based on a specification (the referencing specification describes where a coordinate end up on your map, the geometry specification defines what a surface is, that GML specification defines an XML file format for features, and so on...). Basically the model is the same - why can't we share code. 

How can we work together?

The GeoAPIproject defines common interfaces for much of the middle ground. It even has a procedure for projects to collaborate together. Here is an example: that idea of "Referencing" has a set of interfaces, and there are two implementations: If you are building an application you can pick and choose. It would be great if we could pull this off for some more concepts. The process is in place, but perhaps our approach (our sales pitch) has been wrong. 

Why don't we work together?

One (bad) design trick you can play to get collaboration started is to present a design that is mostly right, and let others find and correct the mistake. By getting everyone to contribute it becomes much more of a shared property and of interest to all. Currently we have been driving GeoAPI from mostly one view point - the specifications. This is easy to understand - since the specifications have all the hard naming choices figured out. However by starting with a specification, the process does not feel "open" to collaboration. In practise the specifications always have mistakes, and collaboration is vital for success. I just wish I could make that point more: Got Java? Got Geography? GeoAPI needs you! I wish we could stress community building as much as correctness. Here is where our approach can be fixed: by saying "We should work together" we are being too timid - the statement is easy to agree to - but chances are it will never happen. Enthusiasm, funding and deadlines drive software here rather than good intentions. 

We should work together on THIS?

So here is the statement I would like us to make (rather than working on something "abstract"). Let's work together on something specific. What is a Map I just want to capture the data model here, we can bust out different widgets to visualize the thing after. A couple of suggestions:
  • Have a look around and see what projects do in the wild
  • GeoTools is sick of its representation (it does not do WMS layers) and would love to collaborate on this
  • OpenJUMP may be in a similar situation?
  • GeoWidgets would love to be the keeper of a Map widget
  • OSGeo has a a SoC student working on a 3D widget
If we really get stuck on names - the file format for Open Web Context Document is around. I want to read in and display the result in a number of widgets. The file format is not finished yet (so we provide feedback if we see something we hate). Treat this as an opportunity not a constraint. Sample code exists in uDig to read the file format. All the best and happy hacking.  
What a silly situation performance (and benchmarks are) have put us in. As developers we really want to know how well things will perform - even if it is just so we can figure out how much hardware we will need at the end of the day. I first ran into this problem with PostGIS (a set of spatial extensions for the PostgreSQL database). The oracle spatial license does not let you run benchmarks and publish. So the only thing I can do is tell you "it is really good". PostGIS is such a success that most spatial data wranglers check it out, and measure for themselves, before going further with Oracle. Oracle does do somethings very well, the security model is awesome. With their "Free for Development" download they are even getting more open source love ... but until we can place that jdbc driver in a maven repository there will not be very many turn key open source oracle solutions. 

FOSS4G Presentation Material

In todays GeoServer IRC meeting the subject of presentations for the Free And Open Source Software for Geoinformatics was on the agenda. One thing the community is really proud of is how much GeoServer has improved in the last year. The community has been really brave, rewriting their raster support to the point it shines, and putting together a really nifty dispatch system in order to take on the monster that is WFS 1.1. Of course all of that is technical and does not make a good presentation (except to other developers so it will be a good beer garden subject). What does make a good presentation isperformance. And this is where open source volunteers shy away from the legal mess that is benchmarking. Specifically we can set up a nice chart comparing the usual open source subjects. This mostly means MapServer since we enjoy talking to Frank about GDAL performance. But for the commercial offerings? Like a comparison with ArcIMS? We did think of a couple of ways: 
  • Report the performance of some publicly available ArcIMS instance
  • Run our performance test during the presentation (it would not be on the slides but the audience could see)
Sadly it will not make the cut, life is too short to figure out what we are and are not allowed to say. 

So what can you tell me? How good is GeoServer?

So here is the best idea we can offer you - if you want to know how insanely great GeoServer has become download the (open source) tool we used for testing and measure it yourself. And just because we want you to come to FOSS4G - please attend our presentation (fingers crossed that it will be accepted) and learn where to download said performance testing tool. You could also engage one of several consulting companies to produce a report for you. If you do find out how good GeoServer is keep it on the hush hush.  
Part three on the road to support reprojection as part of the GeoTools Query API, now that the FeatureType is all straigtened out lets change the actual information to match.

To Review

The GeoTools API has fallen into a trap where we promissed to reproject data when asked the correct Query. Problem is none of the implementations picked up on this change. Here is what we want to make possible.
    
CoordinateReferenceSystem world = CRS.decode("EPSG:4326"); // world lon/lat
CoordinateReferenceSystem local = CRS.decode("EPSG:3005"); // british columbia
      
FeatureSource road = store.getFeatureSource( "road" );

DefaultQuery query = new DefaultQuery( "road", Filter.INCLUDE, new String[]{ "geom", "name" } );     
query.setCoordinateSystem( local );
query.setCoordinateSystemReproject( world );
              
FeatureCollection features = road.getFeatures( query );

Since this is needed functionality the developer community quickly hacked around the problem, leaving us with several working examples to grab the right answer from.

Here is the "fix" for reprojecting the data:
    
if (forcedCS != null)
    results = new ForceCoordinateSystemFeatureResults(results, forcedCS);
if (reprojectCS != null)
    results = new ReprojectFeatureResults(results, reprojectCS);

The Review

In part one we applied the example that changed the FeatureType, now we need to look at reprojecting the data.

In part two we got distracted by some inconsistent error handling ... and made a choice between fail siliently, fail with an exception and doing the right thing. In this case it was easy - the query is experssed using set theory and we made a nice empty set.

Returning to our origional problem  we had test failure when checking to see that the FeatureType in fact changed - turns out I was testing the wrong thing:
  • FeatureCollection.getFeatureType() - returns the type of a the collection. Surprisingly collections are features as well. Makes sense if you consider that a feature is something that can be drawn on a map.
  • FeatureCollection.getSchema() - is the type of the contents of the collection
Bah! When testing the FeatureCollection.getSchema() we found that that our FeatureType has indeed changed.

Looking at the Solution

The two utility classes employed by the "fix" are pretty straight forward:
  • ForceCoordinateSystemFeatureResults - this one just changes the FeatureType ... something we have already accomplished.
  • ReprojectFeaureResults - takes an existing FeatureCollection and grinds through the data applying a MathTransform to each Geometry
Lets look at the part that does the magic: ReprojectFeatureIterator
    
Feature next = reader.next();
Object[] attributes = next.getAttributes(null);

for (int i = 0; i < attributes.length; i++) {
    if (attributes[i] instanceof Geometry) {
        attributes[i] = transformer.transform((Geometry) attributes[i]);
    }
}
return schema.create(attributes, next.getID());

Well that is pretty clear; grab the next Feature suck the attributes out into an array. For each Geometry use a MathTransform to reproject. And then use the FeatureType to create a new Feature with the results.

The MathTransform is produced using a utility class:
    
this.transform = CRS.findMathTransform(originalCs,destinationCS, true);

It sure would be nice if implementors made an optimized version avaialble, I will do this for PropertyDataStore as the last step.

Applying the Solution .. to AbstractFeatureSource? No

So we *could* just apply the wrapping classes as is  ...
    
public FeatureCollection getFeatures(Query query) throws IOException {
        FeatureType schema = getSchema();      
         String typeName = schema.getTypeName();
      
         if( query.getTypeName() == null ){ // typeName unspecified we will "any" use a default
            DefaultQuery defaultQuery = new DefaultQuery(query);
            defaultQuery.setTypeName( typeName );
        }
        else if ( !typeName.equals( query.getTypeName() ) ){
            return new EmptyFeatureCollection( schema );
        }
        FeatureCollection collection = new DefaultFeatureResults(this, query);
        if( collection.getDefaultGeometry() == null ){
            return collection; // no geometry no reprojection needed
        }      
         if ( query.getCoordinateSystem() != null ){
            collection = new ForceCoordinateSystemFeatureResults(collection, query.getCoordinateSystem() );      
         }
        if ( query.getCoordinateSystemReproject() != null){
            collection = new ReprojectFeatureResults(collection, query.getCoordinateSystemReproject() );                      
         }
        return collection;
     }
That would work we would be done .. or would we?

Since we changed the FeatureType of DefaultFeaureResults (to reflect what should be happening) the traditional wrapping approach would be broken. When the ReprojectFeatureResults wrapper goes to produce a math transform it will see the expected CRS already ... and do nothing!

Since DefaultQueryResults is accepting a query I would like to see it do the work, let's get started.

Applying the Solution to DefaultFeatureResults

We need to pick up the correct MathTransform in the constructor (if we cannot do the work we may as well let them know early). Here is the code snip:
    
CoordinateReferenceSystem origionalCRS = origionalType.getDefaultGeometry().getCoordinateSystem();
        if( query.getCoordinateSystem() != null ){
            origionalCRS = query.getCoordinateSystem();
        }
        try {
            transform = CRS.findMathTransform( origionalCRS, cs, true);
        } catch (FactoryException noTransform) {
            throw (IOException) new IOException("Could not reproject data to "+cs).initCause( noTransform );
        }

We are going to use the utility class GeometryCoordinateSequenceTransformer mentioned above to do the actual work.

The next part is to look at the code that does the reading .. getReader():
    
public FeatureReader reader() throws IOException {
        FeatureReader reader = featureSource.getDataStore().getFeatureReader(query,
                getTransaction());
      
         int maxFeatures = query.getMaxFeatures();
        if (maxFeatures == Integer.MAX_VALUE) {
            return reader;
        } else {
            return new MaxFeatureReader(reader, maxFeatures);
        }
    }

Oh look they already have a "wrapper" in place here ... MaxFeatureReader will cut off the feature supply when a specific max is reached.

Lets go shopping:
  • ReprojectFeatureIterator ... utility class used when your collection is memory based
  • ReprojectFeatureReader ... perfect!
Applying this is straight forward:
    
public FeatureReader reader() throws IOException {
        FeatureReader reader = featureSource.getDataStore().getFeatureReader(query,
                getTransaction());
      
         int maxFeatures = query.getMaxFeatures();
        if (maxFeatures != Integer.MAX_VALUE) {
            reader = new MaxFeatureReader(reader, maxFeatures);
        }      
         if( transform != null ){
            reader = new ReprojectFeatureReader( reader, schema, transform );
        }
        return reader;
    }

That should be it ... getBounds uses reader() and so on.

So we have two solutions:
  • An example of how to support reprojection directly as part of your FeatureCollection (ie use of ReprojectFeatureReader or ReprojectFeatureIterator wrappers)
  • An example of how to support reprojection after the fact .. modify your getFeatures( Query method ) to  use ReprojectFeatureResults (or its colleciton based ReprojectingFeatureCollection)
It is very cool to get the reprojection as close to the data read as possible, creating Features, unpacking the attributes to reproject and then folding them back into Features again is a bit of a pain.

Almost done

Tomorrow we will try testing and see what sorts of problems this occurs in two real applications. If there is time we will review the javadocs that started this problem ... and explore ways of preventing this in the future.
One of the questions that came up in yesterdays debug hunt was this ... "what to do when the query does not match the data". Lets see what the code has to say.

We have a couple of options on the table, and actually all of them are implemented! While this is horrible for client code (they cannot know what to expect) it is great for us since we will have working tested code snipets to choose from when we decide what behavior is woth keeping.

To review this problem (pretending we are using Query to access a database) the client code has asked for all the "Rivers" to be returned, but by mistake they asked the "Roads" table.

What to do?

Ignore the Problem

Always an option, sometimes this is phrased as "do what I mean not what I say", by convention code is supposed to LOG a warning when this occurs.

So when the user asked all the "rivers" in the "roads" table ...  we should actually change their query to "roads".

Here is what that looks like (from yesterdays debugging session):
    
public DefaultFeatureResults(FeatureSource source, Query query) {
    this.featureSource = source;      
     FeatureType origionalType = source.getSchema();
  
     String typeName = origionalType.getTypeName();      
     if( typeName.equals( query.getTypeName() ) ){
        this.query = query;
    }
    else {
        this.query = new DefaultQuery(query);
        ((DefaultQuery) this.query).setTypeName(typeName);
    }
    ... handle query
}

I hate things that fail siliently in the night (geotools policy is to produce a warning ... it just has not followed here yet).

Exceptionally Annoying

The next option is to throw an exception ... this is a popular approach (implementors no longer have to worry about the problem, it is visible to client code, and if the exception message is good developers can figure out what went wrong and try again).

This is however an approach we are trying to move away from ... for two reasons:
  • Failure during rendering sucks; you get an empty screen. Much nicer to see the data that worked ... and then sift through the LOG messages to discover why your rivers did not show up.
  • Failure during long running process hurts; GIS data is large usually you will need to try
This approach is taken by the WFSFeatureSource:
    
public FeatureCollection getFeatures(Query request) throws IOException {
    String typeName = featureType.getTypeName();

    if ((request.getTypeName() != null) && !typeName.equals(request.getTypeName())) {
        throw new IOException("Cannot query " + typeName + " with:" + request);
    }
    if (request.getTypeName() == null) {
        request = new DefaultQuery(request);
        ((DefaultQuery) request).setTypeName(featureType.getTypeName());
    }
    return new JDBCFeatureCollection(this, request);
}

Or be correct

The last option, and the one suggested by Andrea is to do what the user said ... if they asked for all the rivers in the roads table give it to them! The collection will however be empty.

This is easy enough to arrange .. from the final AbstractFeatureSource:
    
public FeatureCollection getFeatures(Query query) throws IOException {
    FeatureType schema = getSchema();      
    String typeName = schema.getTypeName();      
      
    if( !typeName.equals( query.getTypeName() ) ){
        return new EmptyFeatureCollection( schema );
    }
    else {
        return new DefaultFeatureResults(this, query);  
    }      
}

I like it.

How Query is Handled

Here is the final result .. I have stolen code from all three examples to cover such matters as a *null* typeName.
    
public FeatureCollection getFeatures(Query query) throws IOException {
    FeatureType schema = getSchema();      
    String typeName = schema.getTypeName();
  
    if( query.getTypeName() == null ){ // typeName unspecified we will "any" use a default
        DefaultQuery defaultQuery = new DefaultQuery(query);
        defaultQuery.setTypeName( typeName );
    }
  
    if( !typeName.equals( query.getTypeName() ) ){
        return new EmptyFeatureCollection( schema );
    }
    else {
        return new DefaultFeatureResults(this, query);  
    }      
}

Thanks for the idea Andrea!  
jive

The code always knows Blog

Posted by jive Feb 20, 2007
Abstract: For the last couple of months the GeoTools community has been creeping  around a problem ... one of quality. I am going to wade in and do something about it; a code reivew. The catch? It is probably going to be my own code.

The problem? We have some plugin implementations that do not follow a coding contact. Normally this is not such a big deal (you throw those plugins out of the build until they shape up). Problem is this time it is *every* plugin.

Unimplemented functionality - how did this Happen?

When this happens it usually means that we (the GeoTools community) have let a user request change our api (probably because some functionality is commonly requested), but we did not take the trouble to get buy in (and time) from the plugin implementors.

Without client code around at the same time as implementations are being built this kind of thing falls through the cracks (release early release often is the mantra, feedback early when developers care is the reality).

The DataStore API has suffered this same fate before; currently uDig only supports three DataStores. As a client application needs events to be fired when editing; but since uDig arrived on the scene a bit late later then the implementations it was not around to ensure this functionality worked as advertised.

Unimplemented functionlaity - why Fix it?

A really easy alternative is to just remove the functionality (it is not being implemented that should tell us something is in a WONT FIX state).

Here is why I am not going to do that ... hacking around this problem is producing the same boiler plate code in a number of spots:
  • In our Renderer .. I saw this back in November when we hooked up Expression (a query langague) so that POJOs (ie normal objects) could be drawn onto a map
  • In other applications .. GeoServer noticed this problem and talked about it in last weeks IRC meeting
  • In other applications .. uDig developers noticed and hacked around the problem so quickly they forgot to report it
It is good practice to only allow a hack three times; and it looks like we are at that number. Since we have three working hacks to draw on the fix should be easy, and removing the hacks and running the test cases should provide us confidence that we have done a good job.. 

Broken - using Query to Reproject

This is what a simple data access query looks like:
    
FeatureSource source = dataStore.getFeatureSource( "road" );
FeatureCollection features = source.getFeatures();

And here is one that uses a Query:
    
FeatureSource source = dataStore.getFeatureSource( "road" );
DefaultQuery query = new DefaultQuery( "road, Filter.INCLUDE );
FeatureCollection features = source.getFeatures();

And finally here is the problem - using Query to ask for something in another projection:
    
CoordinateReferenceSystem world = CRS.decode("EPSG:4326"); // world lon/lat
CoordinateReferenceSystem local = CRS.decode("EPSG:3005"); // british columbia
      

FeatureSource road = store.getFeatureSource( "road" );
DefaultQuery query = new DefaultQuery( "road", Filter.INCLUDE, new String[]{ "name" } );
      
query.setCoordinateSystem( local ); // FROM, optional, will ignore actual data and force CRS
query.setCoordinateSystemReproject( world ); // TO, optional, will reproject data into CRS
              
FeatureCollection features = road.getFeatures( query ); // will reproject the features      

You can see why this is a popular request - it actuall does something a bit more then just data access.
  • setCoordinateSystem() - changes the "Metadata", the information will be returned as is (same values) but the meaning will be changed - the FeatureType will report back that the information is in the provided CRS
  • setCoordianteSystemReproject() - changes the actual "Data", in addition to changing the FeatureType the actual data values will be reprojected and the result returned

Solution? The code always knows...

The needed code is around, it has just not been hooked up behind that getFeatures( query ) method.
Here is the first HACK needed to set the FeatureType up correctly (from DefaultView.java)::
    
FeatureType origionalType = source.getSchema();

CoordinateReferenceSystem cs = null;
if (query.getCoordinateSystemReproject() != null) {
    cs = query.getCoordinateSystemReproject();
} else if (query.getCoordinateSystem() != null) {
    cs = query.getCoordinateSystem();
}
schema = DataUtilities.createSubType(origionalType, query.getPropertyNames(), cs, query.getTypeName(), null);

Here is the second HACK,  the utility classes to do the hard work are right there, they just have not been connected.
    
if (forcedCS != null)
    results = new ForceCoordinateSystemFeatureResults(results, forcedCS);
if (reprojectCS != null)
    results = new ReprojectFeatureResults(results, reprojectCS);

So the solution is to "wrap" the origional feature collection in a helper class that does the work. You can see these same utility classes used in the Renderer, and in GeoTools and in ... your application. (that is the nature of a workaround).
 

First of All the Test Case

In order to debug a problem you need to be able to reproduce it.  Here in java land that tends to mean a test case, for bonus points it means the failure will have a hard time coming back from the dead.

I am going to pick on a *really simple* data store that uses Java property files, simply because I am the implementor responsible for it and I feel guilty :-)

Here is the test case:
    
    public void testQueryReproject() throws Exception {
        CoordinateReferenceSystem world = CRS.decode("EPSG:4326"); // world lon/lat
        CoordinateReferenceSystem local = CRS.decode("EPSG:3005"); // british columbia
       
        FeatureSource road = store.getFeatureSource( "road" );
        FeatureType origionalType = road.getSchema();
       
        DefaultQuery query = new DefaultQuery( "road", Filter.INCLUDE,
                new String[]{ "name" } );
       
        query.setCoordinateSystem( local ); // FROM
        query.setCoordinateSystemReproject( world ); // TO
               
        FeatureCollection features = road.getFeatures( query );
        FeatureType resultType = features.getFeatureType();
               
        assertNotNull( resultType );
        assertNotSame( resultType, origionalType );
       
        GeometryAttributeType resultGeometryType = resultType.getDefaultGeometry();
        assertEquals( world, resultGeometryType.getCoordinateSystem() );
    }

Using a Debugger with Binary Search

This technique works like peanut butter and chocolate.

The first trick is to set a debugger break point somewhere in your code at the "half way point" and make sure you see what you expected to see.

After a bit of a wild ride (through some AbstractDataStore code) turns out that that the DefaultFeatureResults class is going to be doing most of the work.

DefaultFeatureResults Constructor

So let's make sure that we got the Query where we need it - Here is what the code looks like:
    
public DefaultFeatureResults(FeatureSource source, Query query) {
    this.featureSource = source;
    String typeName = source.getSchema().getTypeName();

    if( typeName.equals( query.getTypeName() ) ){
        this.query = query;
    }
    else {
        this.query = new DefaultQuery(query);
        ((DefaultQuery) this.query).setTypeName(typeName);
        ((DefaultQuery) this.query).setCoordinateSystem(query.getCoordinateSystem());
        ((DefaultQuery) this.query).setCoordinateSystemReproject(query.getCoordinateSystemReproject());

    }
}

This looks "odd" if they query type name (say "roads" equals the feature souce typeName then we use the query as is. Okay fine). The odd part is the "else" statement it should really throw an error (we are being asked to look for some content, say "rivers", that we do not have!).  The code goes on to copy the CRS information we are interseted in, but does not pay attention to things like requested attributes ... oh wait it does (they Query was passed in as a constructor argument my bad).

If we want to know who did this work we can use "svn blame":
    
cholmesny     public DefaultFeatureResults(FeatureSource source, Query query) {
cholmesny         this.featureSource = source;
     jive         String typeName = source.getSchema().getTypeName();
  groldan
     jive         if( typeName.equals( query.getTypeName() ) ){
  groldan             this.query = query;
     jive         }
     jive         else {
    aaime             this.query = new DefaultQuery(query);
    aaime             ((DefaultQuery) this.query).setTypeName(typeName);
    aaime             ((DefaultQuery) this.query).setCoordinateSystem(query.getCoordinateSystem());
    aaime             ((DefaultQuery) this.query).setCoordinateSystemReproject(query.getCoordinateSystemReproject());
     jive         }
cholmesny     }

Looks like our friend Andrea was working on this .. lets check to see if it was needed
    
public DefaultQuery(Query query) {
      this(query.getTypeName(), query.getNamespace(), query.getFilter(), query.getMaxFeatures(),
          query.getPropertyNames(), query.getHandle());
      this.sortBy = query.getSortBy();
      this.coordinateSystem = query.getCoordinateSystem();
      this.coordinateSystemReproject = query.getCoordinateSystemReproject();
    }
Looks like this is redundent; coordinateSystem and coordinateSystemReproject are getting changed twice! We can ask Andrea if he was ignoring the TypeName on purpose (or if it was mistake), While we wait to hear back lets have a look at the state of our object.

Looks like the query made it this far okay.
qa1.PNG 

DefaultFeatureResults getSchema()

The first hack is about making sure the correct FeatureType is produced ... lets set a break point and see if DefaultFeatureResults.getSchema() is up to the task.

Here is what the code looks like - does not look like CRS is mentioned at all!
    
public FeatureType getSchema() {
        if (query.retrieveAllProperties()) {
            return featureSource.getSchema();
        } else {
            try {
                return DataUtilities.createSubType(featureSource.getSchema(),
                    query.getPropertyNames());
            } catch (SchemaException e) {
                return featureSource.getSchema();
                //throw new DataSourceException("Could not create schema", e);
            }
        }
    }

Well we can cut and paste the working code in from DefaultQuery (remember the Hack example?). However since this work is going to be *the same* every time we run it - we may as well place the work into the consructor and have it called once.

Munching up the various examples we end up with:
    
public DefaultFeatureResults(FeatureSource source, Query query) {
    this.featureSource = source;      
     FeatureType origionalType = source.getSchema();
  
     String typeName = origionalType.getTypeName();      
     if( typeName.equals( query.getTypeName() ) ){
        this.query = query;
    }
    else {
        this.query = new DefaultQuery(query);
        ((DefaultQuery) this.query).setTypeName(typeName);
    }
    CoordinateReferenceSystem cs = null;      
     if (query.getCoordinateSystemReproject() != null) {
        cs = query.getCoordinateSystemReproject();
    } else if (query.getCoordinateSystem() != null) {
        cs = query.getCoordinateSystem();
    }
    try {
        if( cs == null ){
            if (query.retrieveAllProperties()) { // we can use the origionalType as is              
                 schema = featureSource.getSchema();
            } else {
                 schema = DataUtilities.createSubType(featureSource.getSchema(), query.getPropertyNames());                  
             }
         }
        else {
            // we need to change the projection of the origional type
            schema = DataUtilities.createSubType(origionalType, query.getPropertyNames(), cs, query.getTypeName(), null);
        }
    }
    catch (SchemaException e) {
        LOGGER.log( Level.WARNING, "Could not change projection to "+cs, e );
        schema = null; // client will notice something is amiss when getSchema() return null
    }
}


Not something to be proud of:
  • That SchemaException is being gobbled up; the origional code just "faked it" by using the origionalSchema; this would result in data being returned, with no indication that it was not as asked for!
What does the debugger say? The debugger says that the resulting schema has a "null" GeometryAttribute! Looks like our stop tomorrow will be in DataUtilities createSubType.  
I have had to field this question a couple times in the last week; most recently from a old GeoTools buddy by the name of Cameron. Apparently I am working on Open Location Services and he knows a buddy who is working on the same thing. How would you answer a question like that? We all know the default answers, they form mantras for the open source movement. Things like many eyeballs just don  
At the FOSS4G conference I was charmed to see a bunch of cross project hacking ... in the form of a meeting about "Tiling". The result of this meeting is just now been made available:  The origional that I linked to earlier is here: Tile Map Service Specification (thanks to Schyler for the correction). While this is an exciting concept, the excitement has spread to the media - in this case Directions Magazine: OSGeo Offers Up a Tiling Specification The article explores the idea of OSGeo as a standards body, and I thought I may offer a clarification in this specific instance. The tiling "specification" is currently a bunch of hackers bashing away at the problem. Standardization comes later, the document will be submitted to the OGC where I imagine it will be treated as an extension to WMS (much like the SLD specification). It should be noted that this has been done carefully at a technical level, the requests made conform to the WMS specification. It was very tempting to do something closer to nasa worldwind where tiles are asked for row/col. I always respect a standard that is backed by an implementation, more so when several implementations are available. The fact that OSGeo plays hosts to open source projects, that by definition need to work together, provide a great environment for collaboration in this manner. This is the same approach used by the OGC for their interoptability experiments. Please view this tiling experiment in the same manner, the documents are not offical but we may learn enough to inform a standard. Aside: The OGC does provide a nice venue for "Open standards"; personally I find their direct use of ISO standards is starting to "close" the door. A house with an open door, does no good if the front gate is locked.  

Filter Blog

By date: