PerryGeo

Occasionally I find myself in the C#/.NET world in order to write code using ESRI ArcObjects. Today I was toying with the idea of automating the creation of ESRI Layer files (a file which defines the cartographic styling of a dataset). Of course they are in an undocumented binary file format, inaccessible to anything but ESRI software. So I pop open Visual Studio ….

I feel a nagging unease every time I type a set of curly braces. And VB just makes me insane. I prefer, of course, to use python. Luckily there is IronPython which runs on .NET - which means I could theoretically use it to interact with ArcGIS.

I only found a single working example of using ArcObjects through IronPython. But it looked promising enough to close Visual Studio and give it a go.

The first nagging problem is an IronPython-specific one. Relatively minor annoyance but you have to add the reference to a .NET assembly (library) before you can load it.

import clr
clr.AddReference('ESRI.ArcGIS.System')
clr.AddReference('ESRI.ArcGIS.Carto')
from ESRI.ArcGIS import esriSystem
from ESRI.ArcGIS import Carto

Now there is the issue of grabbing an ESRI license. A little verbose IMO but it could easily be encapsulated in a helper function to clean things up.

aoc = esriSystem.AoInitializeClass()
res = esriSystem.IAoInitialize.IsProductCodeAvailable(aoc,
         esriSystem.esriLicenseProductCode.esriLicenseProductCodeArcView)
if res == esriSystem.esriLicenseStatus.esriLicenseAvailable:
    esriSystem.IAoInitialize.Initialize(aoc,
      esriSystem.esriLicenseProductCode.esriLicenseProductCodeArcView)

Now that we’ve satisfied the demands of our proprietary license overlords, we can proceed with the real work .. in this case I just want to open an existing Layer file and see if the resulting object knows it’s own file path. Really simple, right?

lyr = Carto.LayerFileClass()
if "Open" in dir(lyr): print "The Layer object has an Open method but...."
lyr.Open('C:\\test.lyr')
print lyr.Filename

The Layer object has an Open method but....
Traceback (most recent call last):
 File "“, line 1, in 
AttributeError: ‘GenericComObject’ object has no attribute ‘Open’

Hrm. Looks like we’ve run across bug 1506 which doesn’t allow access to the properties and methods of a given instance - instead your have to work through the functions provided by the implementation. Grr…

Carto.ILayerFile.Open(lyr, 'C:\\test.lyr')
print Carto.ILayerFile.Filename.GetValue(lyr)

That is unwieldy, ugly and unpythonic. What’s the point of object oriented programming if you can’t access the methods and properties of an object directly? Since all ArcObjects applications are based on extending COM interfaces, this would be a major pain in any non-trivial application. Basically, until these .NET-accessible COM objects can be treated in a pythonic way, I don’t see any compelling reason to pursue IronPython and ArcGIS integration. Looks like its back to C# for the moment … (/me take a deep sigh and opens Visual Studio) … unless of course anyone has some brilliant solution to share!!

Another disastrous consequence of inaccurate spatial information… Not only can you accidentally tag your neighbor as a criminal, now it appears that sloppy spatial data has lead to the wrong house getting demolished.

I’ve asked it before but its worth repeating … with all the recent advances in spatial data publishing, where are the advances in metadata and data quality assurance? How do you know where the data comes from, what’s been done to it and by whom? What is the intended use of the data? For the vast majority of the data being shoved out onto the web, these bits of metadata are sorely lacking.

Of course this case is more a matter of one person’s sheer stupidity; I’m not sure any caveats in the metadata would have stopped the wrecking ball!

Dealing with corrupted shapefiles can be a painful experience: programs crash for seemingly no reason, attribute tables get screwy, features get lost, queries results don’t look right and ArcGIS processing tools fail with mysterious error codes:

Never fear, OGR is here. The magic bullet for fixing corrupted shapefiles is, 90% of the time, accomplished by using ogr2ogr to convert the shapefile to another shapefile.

ogr2ogr -f “ESRI Shapefile” shiny_new_clean_dataset.shp corrupted_dataset.shp corrupted_dataset

OGR’s internal data model cleans it up and the output is a fresh shiny new shapefile that works without hassle.

Real-time spatial application developers take note…

I’ve been following the Tour of California this week (looking forward to the Solvang Time Trial this Friday) and have been disappointed with the TV coverage on Versus. Its not that the coverage is bad, its just that long-distance endurance sports don’t lend themselves to the traditional 2 announcers and 1 camera format. There are multiple groups of riders and so much spatial information to keep track of if one really wants to understand the dynamics of a cycling event.

Maybe I’ve just been spoiled by the Amgen Tour Tracker. It is a crowning example of a spatially-aware real-time web application.

It provides two cameras of live coverage, live commentary with interviews, chat, summary updates, gps tracking of riders shown on both an elevation profile and a yahoo-based aerial map, “gps+” location prediction, race standings, time checks, etc. Far more information than any TV coverage without resorting to information overload.

Last time I posted on this blog, Hillary and Obama were still battling it out for the Democratic nomination. Now Barack Obama is our president with an uphill battle to save the economy. So yeah, it’s been a while. I haven’t been doing too much innovative Geo-related stuff lately, hence the lack of blog posts. I’ll try to pick up the pace a bit, even if I have to resort to fluff pieces like this one…

Well, it looks like the economic stimulus bill is going to pass. The bill doesn’t actually specify the projects that will be funded; the money will be allocated to cities and some federal grant agencies. The mayors have already proposed thousands of “shovel-ready” projects that might get a green light depending on how much funding the city gets.

There’s a great site, stimuluswatch.org, that allows the public to review these proposals. Good to know where our tax dollars are headed!

There are several GIS proposals ranging from projects with specific, well-defined (and measurable) objectives to the nebulous “Give us $500,000 to upgrade our cities’ GIS program”. It will be interesting to see which ones pan out, which ones produce results and which ones are just a pure waste of taxpayer dollars.

P.S. If you’d like to see where most of my time and energy is going these days, it’s training for the US National Cup mountain bike race series. My cycling exploits are available for all who are inclined to read them.

Radiohead realeased their video for House of Cards yesterday. Besides being a big radiohead fan, I was also loving the LIDAR technology behind the video.

If you want to check it out yourself, there are code samples on the site as well as access to the raw data. The csv files have four columns (x, y, z, and intensity). For me the quickest way to visualize the data was through R and it’s OpenGL interface called rgl (which is a wonderful high-level 3D data visualization environment).

Assuming you have R installed, rgl is a simple add on through the CRAN repositories:

install.packages("rgl")

Then you need to load the library, load the csv, scale the intensity values from 0 to 1. Then it’s a simple rgl.points command to get an interactive 3D rendering:

library(rgl)
d < - read.csv("C:/temp/radiohead/22.csv", header=FALSE)

# scale intensity values from 0 to 1
d$int <- d[,4] / 255

# rgl.points(x,y,z,size=__,color=__)
# note y value is inverted
# color is a grayscale rgb based on intensity
rgl.points(d[,1],d[,2]*-1,d[,3], size=3, color=rgb(d$int,d$int,d$int))

That’s all it takes to render Thom Yorke in all his 3D digital glory:

So, despite frustrations with getting submitted URLs to appear, Geospatial Reddit is still puttering along. Not exactly a vibrant community yet but there are currently 133 subscribers. If you’re subscribed, take a minute to submit your favorite URLs. If you haven’t subscribed, check it out.

I thought 133 subscribers was a decent number until I found that the Bacon subreddit has over 500. Apparently the world would rather discuss their greasy breakfast food than maps.

For those of you that use EndNote to keep track of your bibliographies/references , there is an alternative. JabRef. I find the UI to be very intuitive and it has a range of customizable import/export formats. JabRef uses the BibTex format as it’s native file format so, of course, it integrates very well with LaTeX.

One of the neat features is the ability to create custom bibliographies in HTML, complete with javascript-based search capabilities. Here’s my reference list which I’ll be slowly adding to as I convert all my old text-based and EndNote reference lists over.

Some folks have had trouble submitting links so I figured I should post a bit more detail on that. To get articles to show up on the geospatial reddit (not the main reddit), go to http://reddit.com/r/geospatial/submit or click the “Submit a Link” button on the right - from the geospatial page. When you’re submitting the url, you should see “submit to geospatial” as the page header.

I know at least 2 of us have been successful at posting. If this doesn’t work for you, please let me know and I’ll try and figure it out.

All the great GIS news/blog aggregators out there (planetgs, slashgeo, etc) are moderator driven - a few people act as the gatekeepers and inevitably have to decide what information is useful. This is not the ideal way to do things.

There’s a more democratic and distributed way to spread the role - it’s called reddit. More specifically, Geospatial Reddit. For those unfamiliar with reddit (or similar sites like digg), the idea is simple: users submit stories and users vote on stories. The most popular ones rise to the top and, theoretically, the best articles magically appear on the front page. Much like democracy itself, there are flaws in the theory but its the best thing we’ve got.

Geospatial Reddit is public so sign up, submit your favorite stories and vote. Lets see if we can make this work.