PerryGeo

Since April 21, I have not been able to view or extract any data from the USGS Seamless site, ostensibly the central distribution center for the US National Map. The site has been changing rapidly from day to day ever since and it seems that changes are underway so at least we know someone is working on it.. or hacking it to pieces. The last day or two they appear to have given up and are just redirecting people to gisdata.usgs.gov which, of course, has no mention of the outage on the home page.

When I develop an internet application, even if it’s only used by a few people, I usually seperate the development version from the stable, live version to minimize any downtime. And if you absolutely can’t keep the app running, at least put a big banner on the page indicating that the system is down so people (like me) don’t waste half an hour trying to figure out what they’re doing wrong. Is this too much to ask of the USGS? They are supposed to be the official portal for accessing our nation’s spatial data, right? And we’re not talking about a small server hiccup here, it has been down since at least April 21st with no public indication that problems are occuring on the site.

I just recieved an email this morning from the USGS web mapping admin. The emphasis is mine:

We apologize for any issues you may have experienced lately. The Seamless server, and all related map services will be unavailable for at least the next few days. During this time, the sites may still appear to be functioning. Some may ask for a password, and others may not show up at all. Normally our status messages are posted at http://seamless.usgs.gov. However, since this server has been affected by this outage, users are being re-directed to http://gisdata.usgs.net. We are in the process of posting a message here as well, which you will be able to monitor for any updates. We are estimating that the site will be available again by Monday May 1st 2006. Our team is working diligently to have this service available as soon as possible. We appreciate you patience during this time.

I really shouldn’t be surprised that a government agency botched it so badly; that seems to be the norm here in the US. But I’ve really come to rely on the seamless site for alot of data and it seems that 10 days of downtime for the sole distributor of our seamless national spatial data archive is a bit… amateur.

Time and time again I see open source and open standards mentioned in the same sentence. While I’m a strong proponet of both, it is a bit disheartening to see how closely intertwined the two concepts are in the eyes of many GIS folks.

Open source refers to software distributed with a license that allows access to view and modify the source code. There are also some other criteria but unrestricted access to the source code is the key component.

Open standards refers to software-neutral specifications, usually developed collaboratively, to accomplish a technical goal. In the GIS world, this typically means OpenGIS specifications for sharing data across a network (WMS/ WFS/ WCS), data formats (GML), or for working with spatial data in a relational database (Simple Features Spec for SQL). We could arguably include pseudo-open specifications for data such as shapfiles and KML.

Open source applications do not always conform to open standards. Standards-compliant software does not necessarily have to be open source. So why are the two often mentioned in the same breath as though they were synonymous? Perhaps open source software is perceived as being “ahead” of other types of software in terms of adoption of standards; and maybe that’s true. But there are many proprietary software companies that have devoted alot of effort towards making their software communicate via open standards and their efforts should not go unnoticed (ESRI and Cadcorp just to name the two I’m familiar with).

The promise of open standards is that anyone can develop and use compliant applications that can easily interoperate regardless of the chosen software package. While that promise is far from being fully realized, associating open standards with a particular type of software will not get us any closer.

Update: Or maybe we are getting close… check out Geoff Ziess’ post on the OGC interoperability demonstration in Tampa. Ten vendors interoperating and sharing data in real time… this is what it’s all about.

The Cartography blog recently talked about a series of excellent Paleogeographic maps developed by Dr. Ron Blakey at Northern Arizona University. Ever since I first studied geology, I had dreamed of an atlas that would clearly and visually demostrate how our current land masses came to be. This time series of maps focuses on North America and the geologic events that shaped have shaped it for the last 500 million years. Truly fascinating and excellent work. I encourage everyone to check out the site and read a little about it as well as the narrative by Geoff Manaugh .

Now it occured to me that a time series of maps lends itself very well to an animated sequence. While I am no graphic artist, I have done a few projects in the past that required stiching together a time-series of maps into an animated gif. The process is fairly simple:

1. Download or create each map you want to include in the series. For best results, all maps should have the same size and extents.
2. Rename the images in alpha-numeric order (001.jpg, 002.jpg…. 045.jpg)
3. Install ImageMagick - a collection of efficient command line tools for image processing. It supports almost every common image format available these days.
4. run the convert command to create the animated gif:
   

   convert -geometry 500x483 -delay 200 -loop 0 *.jpg mymovie.gif
   

   The geometry is simply the WIDTHxHEIGHT dimensions of the output image (it helps if this is proportional to the original image dimensions).

   The delay parameter specifies how many hundreths of a second delay occurs between each frame.

   The loop parameter, when set to zero, indicates the gif will loop infinitely.

   The *.jpg, if your operating environment supports wildcards, will take each of the jpg images in the current directory and stich them into an animated gif named mymovie.gif

Viola! An animated movie from a series of static maps. In the case of the Paleogeologic maps, there were 41 maps which produced a sizable animated gif (about 7.5 MB). You can check out the results here. I could watch this play for hours!! Really fascinating stuff.. many thanks to Dr. Ron Blakey for putting this project together.

LIDAR data is certainly a hot technology these days. LIght Detection And Ranging data can be used to create extremely detailed terrain models but there are lots of barriers to using LIDAR data effectively. USGS Center for LIDAR Information Coordination and Knowledge was put in place to “facilitate data access, user coordination and education of lidar remote sensing for scientific needs“.

Beyond the sheer size of the datasets and the knowledge and hardware required to process them, software is a big issue. In the realm of open-source GIS tools, there are many applications (GRASS being the most prominent) for dealing with elevation point data and processing it into more meaningful products such as elevation DEMs and contours.

Usually the data comes as simple ASCII text files and the x,y and z values are easily extracted from such a file. But take a look at the USGS data distribution site and you’ll notice some of the datasets are distributed as LAS binary files. It makes sense to store such massive datasets in binary so I started looking for some LAS conversion tools.So after some searching, I found a bunch of proprietary products for working with LAS but no open source tools. Luckily, the format is well documented thanks to the efforts by the ASPRS to make it an open specification.

So dusting off my notes about parsing binary files in python, I set out to create a python module for extracting LIDAR data from LAS files. The LAS format contains a header which needs to be parsed first in order to read the point cloud. Once you have the header info, you can scan your way through the dataset to pick out the x,y,z values.

Here’s an example of the python interface that will read the first 10,000 points into a 2D shapefile with the elevation as a attribute in the dbf:

import pylas
infile = 'sanand000001.las'
outfile = 'lidar.shp'
header = pylas.parseHeader(infile)
pylas.createShp(outfile, header, numpts=10000, rand=False)

The issue I struggled with is the sheer size of these datasets. A USGS quarter quad can contain 10 million points which is an excessive number of points to create, say, a 10 meter DEM over such a small area. Clearly there was a need to extract a subset of this dataset but just taking the points sequentially gives you a subset of the total area. So, by default, pylas randomly scans the data to pull the number of specified points so that the point cloud could cover the entire area (at a much lower point density). Without numpts specified, it will randomly select 1/2000th of the total number.

So the simplified interface to make a more manageable lidar shapefile would be:

header = pylas.parseHeader(infile)
pylas.createShp(outfile, header)

Once the shapefile is created, you can bring it into GRASS to do the processing to generate DEMs, contours and other derived elevation products:

v.in.ogr dsn=lidar.shp layer=lidar output=lidar
g.region vect=lidar
g.region res=10
v.surf.rst input=lidar elev=lidar_dem zcolumn=elev

# Launch the interactive 3D viewer
nviz lidar_dem

Of course the method I just described is very simplistic and does not even come close to utilizing the full potential of the LIDAR point cloud, but it’s a start.

The pylas.py module can be downloaded here. The code has worked for me on the few datasets I’ve tested it with but it should certainly be considered a rough-cut, alpha product. There is much room for improvement and, of course, if you have any suggestions or contributions, please get in touch.